MSBuild-如何并行构建多个文件和项目 [英] MSBuild - How to build multiple files AND projects in parallel

问题描述

我有一个很大的Visual Studio 2010解决方案文件(.sln)，其中包含大量项目(~50).每个项目大约由20，.cpp和.h文件组成.在快速的Intel i7机器上构建整个项目大约需要2个小时，所有依赖项都本地缓存在快速的SSD上.

I have a large Visual Studio 2010 solution file (.sln) that contains a large number of projects (~50). Each project consists of roughly 20 .cpp and .h files. Building the entire project takes roughly 2 hours on a fast Intel i7 machine with all the dependencies cached locally on a fast SSD.

我正在尝试复制我已经在该主题上找到了一个现有实验，并且需要对以下方法进行一些澄清:

I am attempting to replicate an existing experiment I found on this topic, and need some clarification on how to:

1. 获取我要在Visual Studio(使用GUI/IDE)中使用的并行化功能.
2. 通过MSBuild.exe使用自动命令行构建也可以获取这些更改.

1. Get the parallelization I'm seeking to work in Visual Studio (with the GUI/IDE).
2. Get these changes also working with automated command-line builds via MSBuild.exe.

首先，/m和/maxcpucount选项是否相同?我已经阅读了有关/m选项的文章，该文章似乎可以提供更多的功能项目.这似乎与我通过GUI中的以下操作设置的选项相同:

First, are the /m and /maxcpucount options the same? I've read an article on the /m option, which seems to build more projects in parallel. This appears to be the same option I set via the following actions in the GUI:

• 工具
  • 选项
  • 项目和解决方案
  • 构建并运行
    • Maximum number of parallel project builds
    • Tools
      • Options
      • Projects and Solutions
      • Build and Run
        • Maximum number of parallel project builds

        还有另一个选项/MP，我可以通过以下方式在GUI中进行访问:

        There is another option, /MP, which I can reach in the GUI via:

        • 项目
          • 属性
          • 配置属性
          • C/C ++
            • 常规
            • Multi-processor Compilation
            • Projects
              • Properties
              • Configuration Properties
              • C/C++
                • General
                • Multi-processor Compilation

                如果我错了，请纠正我，但这似乎表明该项目将并行构建多个.cpp文件，但是没有选择指定多少文件的选项，除非我必须手动在其中设置.文本框(即/MP 4或/MP4.

                Please correct me if I'm wrong on this, but this seems to suggest that the project will build multiple .cpp files in parallel, but there's no option to specify how many, unless I have to manually set it in the text box (ie: /MP 4 or /MP4.

                为了使/MP正常工作，我似乎需要禁用最小限度的重建(即:Enabled Minimal Rebuild: No (/GM-))，并且它也不适用于预编译的头文件.我通过使预编译头成为该解决方案先于所有其他项目的专用项目，解决了预编译头问题.

                It also seems that in order for /MP to work, I need to disable minimal rebuild (ie: Enabled Minimal Rebuild: No (/GM-)), and it also doesn't work with pre-compiled headers. I've worked around the pre-compiled headers issue by making the precompiled headers be a dedicated project that the solution builds first, before all other projects.

                问题:如何通过解决方案中的并行构建，在解决方案中为并行项目构建实现这些选项，并在项目中构建并行文件(.cpp)，并确认它们是否按预期运行?我上面的假设也正确吗? 我的最终目标是测试构建项目时所使用的内核总数，或者如果我的构建过程是I/O绑定而不是CPU绑定的话，甚至可能使项目超载.在Linux中可以通过以下方式轻松实现:

                To the question: How can I implement these options for parallel project builds within the solution, and parallel file (.cpp) builds within the projects, using MSBuild via the command-line, and confirm that they are working as expected? Are my assumptions above also correct? My end goal is to test building the project with up to the total number of cores being uses, or possibly even overloading it if my build process is I/O bound rather than CPU bound. I can do this easily enough in Linux via:

                NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
                NUMJOBS="$(echo "$NUMCPUS*1.5" | bc)"
                NUMJOBS="$(printf '%.0f' $NUMJOBS)"
                alias pmake='time nice make -j$NUMJOBS'
                

                这使我可以在8核PC上一次将12个构建任务排队，并且make自动并行最多构建12个.cpp文件，处理隐式构建规则/依赖关系等.我试图在Visual Studio Professional和MSBuild中实现相同的目的.当我的构建受I/O约束时，这种方法是使所有内核保持~100%使用率的最佳方法.

                This allows me to, on my 8-core PC, queue up 12 build tasks at once, and make automatically builds up to 12 .cpp files in parallel, handling implicit build rules/dependencies, etc. I'm trying to achieve the same thing in Visual Studio Professional and MSBuild. When my builds are I/O bound, this approach has been the best at keeping all cores at ~100% usage.

                谢谢.

                推荐答案

                编译器(cl.exe)的/MP开关和msbuild.exe的/m选项确实是可行的方法.以下是一些相关的文档摘录:

                The /MP switch for the compiler (cl.exe) and the /m option for msbuild.exe are indeed the way to go. Here are some relevant documentation extracts:

                /MP选项使编译器创建一个或多个副本 本身，每个都在一个单独的过程中.然后这些副本同时 编译源文件.可选参数:最大数量 编译器可以创建的进程.如果省略processMax 参数，编译器将检索有效处理器的数量 您的计算机从操作系统，并创建一个过程 每个处理器. /MP选项与某些编译器选项和语言功能不兼容.如果您将不兼容的编译器选项与/MP选项一起使用，则编译器会发出警告D9030并忽略/MP选项

                The /MP option causes the compiler to create one or more copies of itself, each in a separate process. Then these copies simultaneously compile the source files. Optional Argument: The maximum number of processes that the compiler can create. If you omit the processMax argument, the compiler retrieves the number of effective processors on your computer from the operating system, and creates a process for each processor. The /MP option is incompatible with some compiler options and language features. If you use an incompatible compiler option with the /MP option, the compiler issues warning D9030 and ignores the /MP option

                示例，假设您指定以下命令行:

                Examples suppose you specify the following command line:

                cl/MP7 a.cpp b.cpp c.cpp d.cpp e.cpp

                cl /MP7 a.cpp b.cpp c.cpp d.cpp e.cpp

                在这种情况下，编译器使用五个进程，因为这是 少于五个源文件，最多七个进程. 另外，假设您的计算机有两个有效的处理器，并且 您指定以下命令行:

                In this case the compiler uses five processes because that is the lesser of five source files and a maximum of seven processes. Alternatively, suppose your computer has two effective processors and you specify the following command line:

                cl/MP a.cpp b.cpp c.cpp

                cl /MP a.cpp b.cpp c.cpp

                在这种情况下，操作系统报告两个处理器.所以， 编译器在计算中使用两个过程.结果， 编译器将通过两个进程执行构建，因为这是 少于两个进程和三个源文件.

                In this case the operating system reports two processors; therefore, the compiler uses two processes in its calculation. As a result, the compiler will execute the build with two processes because that is the lesser of two processes and three source files.

                因此，正如您所注意到的那样，这全都与编译器有关，而对于msbuild则一无所知，因为编译器是独立的工具.换句话说:假设您打开2个命令窗口，并且在这两个窗口中同时调用cl /MP3 a.cpp b.cpp c.cpp之类的命令，则您将同时运行6个编译器进程.这是您要遵循的行为，这也是msbuild可以执行的操作.

                So as you noticed this was all about the compiler, and no word about msbuild yet, since the compiler is a standalone tool. In other words: suppose you open 2 command windows and in both of them invoke a command like cl /MP3 a.cpp b.cpp c.cpp at the same time, you'd have 6 compiler processes running at the same time. Which is the behaviour you are after and which is also what msbuild can do.

                调用msbuild /?的输出:

                /maxcpucount [:n]指定并发进程的最大数目 与建立.如果不使用该开关，则使用的默认值为1. 如果使用的开关没有值，则MSBuild将最多使用 计算机上的处理器数量. (简短格式:/m [:n])

                /maxcpucount[:n] Specifies the maximum number of concurrent processes to build with. If the switch is not used, the default value used is 1. If the switch is used without a value MSBuild will use up to the number of processors on the computer. (Short form: /m[:n])

                还有更多相关信息，此处，特别是其中的部分说

                Also more relevant information here, specifically the part where it says

                BuildInParallel是MSBuild任务上的可选布尔参数. 当BuildInParallel设置为true(其默认值)时，多个 生成工作流程以在同一时间构建尽可能多的项目 时间越长越好.为了使其正常工作，请使用/maxcpucount开关 必须设置为大于1的值，并且系统必须至少为 双核或具有两个或更多处理器.

                BuildInParallel is an optional boolean parameter on a MSBuild task. When BuildInParallel is set to true (its default value), multiple worker processes are generated to build as many projects at the same time as possible. For this to work correctly, the /maxcpucount switch must be set to a value greater than 1, and the system must be at least dual-core or have two or more processors.

                现在，如果您只是在一个典型的项目上调用msbuild /m my.vcxproj，这将不会做任何特殊的事情，因为只有一个项目可以构建.但是，如果在msbuild文件中，则会调用MsBuild任务，例如

                Now if you'd just call msbuild /m my.vcxproj on a typical project this won't do anything special because there is only one project to build. However if inside an msbuild file there is an call to the MsBuild task like

                <MsBuild Projects="@(ListOfProjects)" BuildInParallel="True"/>
                

                和将/m选项传递给msbuild，然后msbuild将按照上述规则生成其他多个msbuild进程.碰巧的是，这正是使用msbuild构建解决方案时发生的情况. Msbuild首先将解决方案转换为实际的msbuild文件，该文件会使用以下命令调用MsBuild任务解决方案中的项目列表.这样，可以并行构建多个项目，并为此创建多个msbuild进程.反过来，如果这些是设置了/MP选项的C ++项目，则这些项目中的每个项目都将创建多个编译器进程，从而产生最多<max num cl processes> * <max num msbuild instances>个并行编译.这取决于您的构建机器，最佳的是什么，如果您愿意，可以使用这些选项(要有效地执行此操作，每个项目都将导入一个公共属性表，在其中设置/MP选项，否则您将不得不对其进行编辑分别查看所有内容)，并查看生成完成后msbuild报告经过的时间.我总是只将/m和/MP的参数保留为默认值，因为我不希望其他使用我的项目的开发人员进行可能的次优配置，并且因为它很容易使CPU耗尽.

                and the /m option is passed to msbuild then msbuild will spawn multiple other msbuild processes per the rules laid out above. And coincidently that is exactly what happens when building a solution with msbuild. Msbuild first converts the solution to an actual msbuild file which calls the MsBuild task with a list of projects in the solution. As such multiple projects can be built in parallel and for that multiple msbuild processes are created. In turn, if these are C++ projects with the /MP option set, each of those projects will create multiple compiler processes yielding a maximum of <max num cl processes> * <max num msbuild instances> parallel compilations. It depends on your build machine what is the optimum, if you want you can play with the options (to do this efficiently each of your projects would import a common property sheet in which you set the /MP option else you'd have to edit all of them seperately) and look at the elapsed time msbuild reports when a build is done. I always just leave the arguments for /m and /MP to the default because I don't want other developpers using my projects to have possible suboptimal configurations, and because it easily maxes out CPU.

                这篇关于MSBuild-如何并行构建多个文件和项目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！