GNU make-如何模拟多个同时作业 [英] GNU make - how to simulate multiple simultaneous jobs

问题描述

我知道要允许make是多线程的，我使用命令make --jobs=X，其中X通常等于内核数(或两倍或更多).

I know that to allow make to be multithreaded, I use the command make --jobs=X where X is usually equal to number of cores (or twice that or whatever).

我正在调试一个makefile-实际上由许多makefile组成-以与--jobs=X选项一起使用.这是为什么它当前不起作用的一个示例:

I am debugging a makefile - actually consists of many makefiles - to work with the --jobs=X option. Here's an example of why it currently doesn't:

T1:
  mkdir D1
  output_makefile.bat > ./D1/makefile
T2:
  cd D1
  make

用--jobs=X执行此操作将导致争用条件，因为未将T1指定为T2的依赖项，最终T2将在T1之前构建.我需要修复的大多数错误都是这种错误.

Executing this with --jobs=X will lead to a race condition because T1 is not specified as a dependency of T2 and eventually T2 will get built ahead of T1; most of the bugs I need to fix are of this variety.

如果--jobs=X中的X大于逻辑或物理数量?核心，同时执行的作业数量将以逻辑或物理"数量为上限.核心.

If X in --jobs=X is greater than the number of ?logical or physical? cores, the number of jobs executed simultaneously will be capped at the number of ?logical or physical? cores.

我的机器具有4个物理/8逻辑核心，但是将要运行我们的构建的构建机器将具有多达64个核心.

My machine has 4 physical/8 logical cores but the build machine that will be running our builds will have as many as 64 cores.

所以我担心，仅仅因为我的makefile(a)正确构建了最终输出(b)在我的计算机上使用--jobs=4运行没有错误，并不意味着它会正确运行并且在--jobs=64上运行没有错误64核计算机.

So I'm concerned that just because my makefile (a) builds the final output correctly (b) runs without errors on my machine with --jobs=4 does not mean it'll run correctly and without errors with --jobs=64 on a 64-core machine.

是否有一个工具可以模拟make在内核比物理计算机多的环境中执行?

Is there a tool that will simulate make executing in an environment that has more cores than the physical machine?

如何创建一个具有64核的虚拟机并在我的4核计算机上运行它? VMPlayer甚至允许吗?

What about creating a virtual machine with 64 cores and run it on my 4-core machine; is that even allowed by VMPlayer?

我意识到我对make的理解是不正确的:创建的作业插槽 make等于--jobs=N参数，而不是我的PC拥有的内核或线程数.

I realized that my understanding of make was incorrect: the number of job slots make creates is equal to the --jobs=N argument and not the number of cores or threads my PC has.

但是，这本身并不一定意味着make也会并行执行 那些作业，即使我通过使用task-切换.

However, this by itself doesn't necessarily mean that make will also execute those jobs in parallel even if I have fewer cores than jobs by using task-switching.

我需要确认所有作业都是并行执行的，而不仅仅是排队"并等待活动执行的作业完成.

I need to confirm that ALL the jobs are being executed in parallel vs merely 'queued up' and waiting for the actively executing jobs to finish.

因此，我创建了一个包含16个目标的makefile-超过了我拥有的线程或内核的数量-每个配方仅echos目标的名称具有可配置的次数.

So I created a makefile with 16 targets - more than the num of threads or cores I have - and each recipe merely echos the name of the target a configurable number of times.

make.mk

all: 1 2 3 4 ... 14 15 16

<target X>:
    @loop_output.bat $@

loop_output.bat

loop_output.bat

@FOR /L %%G IN (1,1,2048) DO @echo (%1-%%G)

输出将类似于

(16-1)   <-- Job 16
(6-1400)
(12-334)
(1-1616) <-- Job 1
(4-1661)
(15-113)
(11-632)
(2-1557)
(10-485)
(7-1234)
(5-1530)

格式为Job#X-Echo#Y.我在 (16-1)之后看到(1-1616) 的事实意味着make实际上与目标1同时执行目标16.

The format is Job#X-Echo#Y. The fact that I see (1-1616) after (16-1) means that make is indeed executing target 16 at the same time as target 1.

另一种选择是，make完成作业(1-#个内核/线程)，然后 then 接受另一份等于#num个内核/线程的作业，但这不是正在发生的事情.

The alternative is that make finishes jobs (1-#of cores/threads) and then takes another chunk of jobs equal to #num cores/threads but that's not what's happening.

推荐答案

请参阅我的"UPDATE 1":

See my "UPDATE 1":

不需要特殊软件或make技巧.不管您拥有多少内核，Make都将通过生成多个进程并使OS与其他进程一样多任务来真正并行地执行作业.

No special software or make tricks are required. Regardless of number of cores you have, Make will execute the jobs truly in parallel by spawning multiple processes and letting the OS multitask them just like any other process.

Windows PITFALL#1:SourceForge上可用的Gnu Make版本是3.81，它甚至不能使用--jobs执行.您必须下载4.2版并进行构建.

Windows PITFALL #1: The version of Gnu Make available on SourceForge is 3.81 which does NOT have the ability to even execute using --jobs. You'll have to download ver 4.2 and build it.

>

Windows PITFALL#2:make 4.2源将无法构建，因为VS2008(及更早版本)没有某些标头.解决方法很容易:您必须用等效的宏替换未找到的符号"的调用.当您尝试构建它时，我正在谈论的内容应该很明显. (我忘记了丢失的符号是什么).

Windows PITFALL #2: make 4.2 source will fail to build because of some header that VS2008 (and older) doesn't have. The fix is easy: you have to replace the invocation of the "symbol not found" with its macro equivalent; it should be obvious what I'm talking about when you try to build it. (I forgot what the missing symbol was).

这篇关于GNU make-如何模拟多个同时作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！