使用multiprocessing.Process并发进程数最多 [英] Using multiprocessing.Process with a maximum number of simultaneous processes

问题描述

我有Python代码:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    for i in range(0, MAX_PROCESSES):
        p = Process(target=f, args=(i,))
        p.start()

运行良好.但是，MAX_PROCESSES是可变的，并且可以是1和512之间的任何值.由于我仅在具有8内核的计算机上运行此代码，因此我需要确定是否有可能限制允许同时运行的进程数.我已经研究过multiprocessing.Queue，但它看起来不符合我的需要-也许我对文档的解释不正确.

which runs well. However, MAX_PROCESSES is variable and can be any value between 1 and 512. Since I'm only running this code on a machine with 8 cores, I need to find out if it is possible to limit the number of processes allowed to run at the same time. I've looked into multiprocessing.Queue, but it doesn't look like what I need - or perhaps I'm interpreting the docs incorrectly.

是否可以限制同时运行multiprocessing.Process的次数?

Is there a way to limit the number of simultaneous multiprocessing.Processs running?

推荐答案

使用multiprocessing.Pool可能是最明智的选择，它会根据系统上可用的最大内核数生成一个工作进程池，然后进行基本操作核心成为可用的任务.

It might be most sensible to use multiprocessing.Pool which produces a pool of worker processes based on the max number of cores available on your system, and then basically feeds tasks in as the cores become available.

标准文档中的示例( http://docs. python.org/2/library/multiprocessing.html#using-a-pool-of-workers )显示，您还可以手动设置内核数:

The example from the standard docs (http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers) shows that you can also manually set the number of cores:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
    print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

如果代码中需要的话，知道multiprocessing.cpu_count()方法也可以方便地计算给定系统上的内核数.

And it's also handy to know that there is the multiprocessing.cpu_count() method to count the number of cores on a given system, if needed in your code.

以下一些代码草案似乎适用于您的特定情况:

Here's some draft code that seems to work for your specific case:

import multiprocessing

def f(name):
    print 'hello', name

if __name__ == '__main__':
    pool = multiprocessing.Pool() #use all available cores, otherwise specify the number you want as an argument
    for i in xrange(0, 512):
        pool.apply_async(f, args=(i,))
    pool.close()
    pool.join()

这篇关于使用multiprocessing.Process并发进程数最多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！