Java下的进程创建速度变慢了吗? [英] Slowing process creation under Java?

问题描述

我有一个大堆(最多240GB，但在此执行阶段的大部分时间中为20-40GB范围)，JVM [1]在具有24个核心的服务器上的Linux [2]下运行.我们有成千上万的对象必须由外部可执行文件&然后将这些可执行文件创建的数据加载回JVM.每个可执行文件都会产生大约半兆字节的数据(在磁盘上)，在过程完成后立即读入该数据当然会更大.

I have a single, large heap (up to 240GB, though in the 20-40GB range for most of this phase of execution) JVM [1] running under Linux [2] on a server with 24 cores. We have tens of thousands of objects that have to be processed by an external executable & then load the data created by those executables back into the JVM. Each executable produces about half a megabyte of data (on disk) that when read right in, after the process finishes, is, of course, larger.

我们的第一个实现是让每个可执行文件仅处理一个对象.这涉及到产生的可执行文件是对象的两倍(因为我们调用了称为可执行文件的外壳脚本).我们的CPU利用率开始时会很高，但不一定是100％，然后慢慢恶化.当我们开始测量以查看发生了什么时，我们注意到流程创建时间[3]持续变慢.虽然从亚秒级开始，但最终会增长到一分钟或更长时间.可执行文件的实际处理时间通常不到 10 秒.

Our first implementation was to have each executable handle only a single object. This involved the spawning of twice as many executables as we had objects (since we called a shell script that called the executable). Our CPU utilization would start off high, but not necessarily 100%, and slowly worsen. As we began measuring to see what was happening we noticed that the process creation time [3] continually slows. While starting at sub-second times it would eventually grow to take a minute or more. The actual processing done by the executable usually takes less than 10 seconds.

接下来，我们将可执行文件更改为采用要处理的对象列表，以尝试减少创建的进程数.批量大小为几百(我们当前样本大小的约 1%)时，流程创建时间从大约 2 秒开始.增长到5-6秒左右.

Next we changed the executable to take a list of objects to process in an attempt to reduce the number of processes created. With batch sizes of a few hundred (~1% of our current sample size), the process creation times start out around 2 seconds & grow to around 5-6 seconds.

基本上，为什么随着执行的继续，创建这些流程需要花费这么长时间?

Basically, why is it taking so long to create these processes as execution continues?

[1] Oracle JDK 1.6.0_22
[2]红帽企业Linux高级平台5.3，Linux内核2.6.18-194.26.1.el5#1 SMP
[3]创建ProcessBuilder对象，重定向错误流，然后启动它.

[1] Oracle JDK 1.6.0_22
[2] Red Hat Enterprise Linux Advanced Platform 5.3, Linux kernel 2.6.18-194.26.1.el5 #1 SMP
[3] Creation of the ProcessBuilder object, redirecting the error stream, and starting it.

推荐答案

我的猜测是，如果Java正在使用fork/exec系统调用来生成子进程，那么您可能会遇到fork/exec问题.

My guess is that you MIGHT be running into problems with fork/exec, if Java is using the fork/exec system calls to spawn subprocesses.

通常fork/exec效率很高，因为fork()几乎没有作用-所有页面都是写时复制的.对于非常大的进程(即那些映射了千兆字节的页面的进程)来说，这种情况就不再成立了，因为页表本身需要花费较长的时间来创建-当然，销毁就必须立即调用exec.

Normally fork/exec is fairly efficient, because fork() does very little - all pages are copy-on-write. This stops being so true with very large processes (i.e. those with gigabytes of pages mapped) because the page tables themselves take a relatively long time to create - and of course, destroy, as you immediately call exec.

由于您正在使用大量堆，这可能会影响您.您映射的页面越多，它变得越糟，这可能是导致逐步降低的原因.

As you're using a huge amount of heap, this might be affecting you. The more pages you have mapped in, the worse it may become, which could be what's causing the progressive slowdown.

考虑以下之一:

• 使用posix_spawn，如果libc中的fork/exec未实现posix_spawn
• 使用一个子流程负责创建/获取其他流程；生成一次，并使用一些IPC(管道等)告诉它该怎么做.

注意:这都是猜测.您可能应该做一些实验，看看是否是这种情况.

NB: This is all speculation; you should probably do some experiments to see whether this is the case.

这篇关于Java下的进程创建速度变慢了吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！