为什么在 java compile 上并行执行会随着时间线性增长 [英] why parallel execution on java compile take linear growth in time

问题描述

time javac Main.java                                      --> 0m1.050s
time javac Main.java & javac Main.java                    --> 0m1.808s
time javac Main.java & javac Main.java & javac Main.java  --> 0m2.690s
time javac Main.java & ... 8 time                         --> 0m8.309s

当我们并行运行 javac 命令并且随着 javac 命令的每次增加，~1 sec 被添加到所有 javac 命令来完成.

When we run javac command in parallel and with each increase in javac command ~1 sec gets added for all the javac command to complete.

为什么时间是线性增长的?

所有javac进程在运行时都涉及locks，如果是，如何克服它以免时间线性增长

Is all javac process while running involved in some kind on locks, if yes how to overcome it so as not to have a linear growth in time

PS:我已经在单核机器、双核机器、4核机器上面试过都表现出相同的行为.

PS: I have tried above on single core machine, double core machine, 4 core machine all showed same behaviour.

PS2:环境RedHat7，javac 1.7.0_79

推荐答案

java 编译器已经可以将其工作分配到可用的处理器上，即使只编译单个文件也是如此.因此，您自己并行运行单独的编译器实例不会产生您期望的性能提升.

The java compiler already handles dividing its work across available processors, even when only compiling a single file. Therefore running separate compiler instances in parallel yourself won't yield the performance gains you are expecting.

为了演示这一点，我在名为 Main1.java 的单个文件中生成了一个大型(100 万行，10,000 个方法)java 程序.然后将额外的副本作为 Main2.java 到 Main8.java.编译时间如下:

To demonstrate this, I generated a large (1 million lines, 10,000 methods) java program in a single file called Main1.java. Then made additional copies as Main2.java through Main8.java. Compile times are as follows:

单文件编译:

time javac Main1.java &    --> (real) 11.6 sec

在 top 中观察这个单个文件编译显示处理器使用率主要在 200-400% 范围内(表示多个 CPU 使用率，每个 CPU 100%)，偶尔会在 700% 范围内出现峰值(这台机器上的最大值是 800%，因为有 8 个处理器).

Watching this single file compile in top revealed processor usage mostly in the 200-400% range (indicating multiple CPU usage, 100% per CPU), with occasional spikes in the 700% range (the max on this machine is 800% since there are 8 processors).

接下来，两个文件同时进行:

Next, two files simultaneously:

time javac Main1.java &    --> (real) 14.5 sec
time javac Main2.java &    --> (real) 14.8 sec

所以编译两个只用了 14.8 秒，而编译一个用了 11.6 秒.这绝对是非线性的.在这些运行时查看 top 很明显，每个 Java 编译器一次最多只利用四个 CPU(偶尔会出现更高的峰值).正因为如此，两个编译器运行在 8 个 CPU 上，大多数情况下彼此并行.

So it only took 14.8 seconds to compile two, when it took 11.6 seconds to compile one. That's definitely non-linear. It was clear by looking at top while these were running that again each java compiler was only taking advantage of at most four CPUs at once (with occasional spikes higher). Because of this, the two compilers ran across eight CPUs mostly in parallel with each other.

接下来，四个文件同时进行:

Next, four files simultaneously:

time javac Main1.java &    --> (real) 24.2 sec
time javac Main2.java &    --> (real) 24.6 sec
time javac Main3.java &    --> (real) 25.0 sec
time javac Main4.java &    --> (real) 25.0 sec

好的，我们已经碰壁了.我们不能再对编译器进行并行化处理了.四个文件用了 25 秒，而两个文件用了 14.8 秒.那里有一些优化，但主要是线性时间增加.

Okay, here we've hit the wall. We can no longer out-parallelize the compiler. Four files took 25 seconds when two took 14.8. There's a little optimization there but it's mostly a linear time increase.

最后，八个同时:

time javac Main1.java &    --> (real) 51.9 sec
time javac Main2.java &    --> (real) 52.3 sec
time javac Main3.java &    --> (real) 52.5 sec
time javac Main4.java &    --> (real) 53.0 sec
time javac Main5.java &    --> (real) 53.4 sec
time javac Main6.java &    --> (real) 53.5 sec
time javac Main7.java &    --> (real) 53.6 sec
time javac Main8.java &    --> (real) 54.6 sec

这实际上比线性差一点，因为八个用了 54.6 秒，而四个只用了 25.0 秒.

This was actually a little worse than linear, as eight took 54.6 seconds while four only took 25.0.

所以我认为所有这一切的收获是相信编译器会在尝试优化您在可用 CPU 资源上提供的工作时做得不错，并且尝试手动添加额外的并行化将受到限制(如果有的话)受益.

So I think the takeaway from all this is to have faith that the compiler will do a decent job trying to optimize the work you give it across the available CPU resources, and that trying to add additional parallelization by hand will have limited (if any) benefit.

作为参考，我在 Oracle 的错误数据库中发现了两个关于增强 javac 以利用多个处理器的条目:

For reference, there are two entries I found in Oracle's bug database regarding enhancing javac to take advantage of multiple processors:

• 错误 ID:JDK-6629150 -- 最初的投诉，最终被标记为重复:
• Bug ID:JDK-6713663 -- 建议解决方案，并基于已解决日期"似乎在 2008 年 6 月 12 日添加了 javac 中的多处理器支持.

• Bug ID: JDK-6629150 -- The original complaint, this was eventually marked as a duplicate of:
• Bug ID: JDK-6713663 -- Suggests the resolution, and based on the "Resolved Date" it appears that multi-processor support in javac was added on 2008-06-12.

这篇关于为什么在 java compile 上并行执行会随着时间线性增长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！