与Python中的multiprocessing/pool.map无关的块大小? [英] Chunksize irrelevant for multiprocessing / pool.map in Python?

问题描述

我尝试利用python的池多处理功能.

I try to utilize the pool multiprocessing functionality of python.

独立于我如何设置块大小(在Windows 7和Ubuntu下-后者在下面有4个内核)，并行线程的数量似乎保持不变.

Independent how I set the chunk size (under Windows 7 and Ubuntu - the latter see below with 4 cores), the amount of parallel threads seems to stay the same.

from multiprocessing import Pool
from multiprocessing import cpu_count
import multiprocessing
import time


def f(x):
    print("ready to sleep", x, multiprocessing.current_process())
    time.sleep(20)
    print("slept with:", x, multiprocessing.current_process())


if __name__ == '__main__':
    processes = cpu_count()
    print('-' * 20)
    print('Utilizing %d cores' % processes)
    print('-' * 20)
    pool = Pool(processes)
    myList = []
    runner = 0
    while runner < 40:
        myList.append(runner)
        runner += 1
    print("len(myList):", len(myList))

    # chunksize = int(len(myList) / processes)
    # chunksize = processes
    chunksize = 1
    print("chunksize:", chunksize)
    pool.map(f, myList, 1)

无论我使用chunksize = int(len(myList) / processes)，chunksize = processes还是1，行为都是相同的(如上例所示).

The behaviour is the same whether I use chunksize = int(len(myList) / processes), chunksize = processes or 1 (as in the example above).

是否可以将块大小自动设置为核心数量?

Could it be that the chunksize is set automatically to the amount of cores?

chunksize = 1的示例:

--------------------
Utilizing 4 cores
--------------------
len(myList): 40
chunksize: 10
ready to sleep 0 <ForkProcess(ForkPoolWorker-1, started daemon)>
ready to sleep 1 <ForkProcess(ForkPoolWorker-2, started daemon)>
ready to sleep 2 <ForkProcess(ForkPoolWorker-3, started daemon)>
ready to sleep 3 <ForkProcess(ForkPoolWorker-4, started daemon)>
slept with: 0 <ForkProcess(ForkPoolWorker-1, started daemon)>
ready to sleep 4 <ForkProcess(ForkPoolWorker-1, started daemon)>
slept with: 1 <ForkProcess(ForkPoolWorker-2, started daemon)>
ready to sleep 5 <ForkProcess(ForkPoolWorker-2, started daemon)>
slept with: 2 <ForkProcess(ForkPoolWorker-3, started daemon)>
ready to sleep 6 <ForkProcess(ForkPoolWorker-3, started daemon)>
slept with: 3 <ForkProcess(ForkPoolWorker-4, started daemon)>
ready to sleep 7 <ForkProcess(ForkPoolWorker-4, started daemon)>
slept with: 4 <ForkProcess(ForkPoolWorker-1, started daemon)>
ready to sleep 8 <ForkProcess(ForkPoolWorker-1, started daemon)>
slept with: 5 <ForkProcess(ForkPoolWorker-2, started daemon)>
ready to sleep 9 <ForkProcess(ForkPoolWorker-2, started daemon)>
slept with: 6 <ForkProcess(ForkPoolWorker-3, started daemon)>
ready to sleep 10 <ForkProcess(ForkPoolWorker-3, started daemon)>
slept with: 7 <ForkProcess(ForkPoolWorker-4, started daemon)>
ready to sleep 11 <ForkProcess(ForkPoolWorker-4, started daemon)>
slept with: 8 <ForkProcess(ForkPoolWorker-1, started daemon)>

推荐答案

Chunksize不会影响正在使用的内核数，这是由Pool的processes参数设置的. Chunksize设置您传递给Pool.map的可迭代项的数量，在Pool称为任务"的每个工作进程中一次分配(下图显示了Python 3.7.1)

Chunksize doesn't influence how many cores are getting used, this is set by the processes parameter of Pool. Chunksize sets how many items of the iterable you pass to Pool.map, are distributed per single worker-process at once in what Pool calls a "task" (figure below shows Python 3.7.1).

如果设置了chunksize=1，则只有在完成之前收到的工作后，才能在新任务中为工人进程提供新的项目.对于chunksize > 1，工人在一个任务中一次获取整批物品，完成​​后，如果还有剩余，则获得下一批.

In case you set chunksize=1, a worker-process gets fed with a new item, in a new task, only after finishing the one received before. For chunksize > 1 a worker gets a whole batch of items at once within a task and when it's finished, it gets the next batch if there are any left.

使用chunksize=1一对一分发项目增加了调度的灵活性，同时降低了总体吞吐量，因为滴灌需要更多的进程间通信(IPC).

Distributing items one-by-one with chunksize=1 increases flexibility of scheduling while it decreases overall throughput, because drip feeding requires more inter-process communication (IPC).

在对Pool的chunksize-algorithm的深入分析中，在此，我定义了工作单元，用于将可迭代的一个项目处理为 taskel ，以避免与Pool使用任务"一词的命名发生冲突.一项任务(作为工作单元)由chunksize个任务组组成.

In my in-depth analysis of Pool's chunksize-algorithm here, I define the unit of work for processing one item of the iterable as taskel, to avoid naming conflicts with Pool's usage of the word "task". A task (as unit of work) consists of chunksize taskels.

如果您无法预测任务需要完成多长时间(例如优化问题)，则设置chunksize=1.此处滴灌可防止工人流程坐在一堆未接触的物品上，而在一个沉重的任务板上el缩时，可防止任务中的其他物品分配到闲置的工人流程中.

You would set chunksize=1 if you cannot predict how long a taskel will need to finish, for example an optimization problem, where the processing time greatly varies across taskels. Drip-feeding here prevents a worker-process sitting on a pile of untouched items, while chrunching on one heavy taskel, preventing the other items in his task to be distributed to idling worker-processes.

否则，如果所有任务组都需要相同的时间才能完成，则可以设置chunksize=len(iterable) // processes，这样任务就只能在所有工作进程中分配一次.请注意，如果len(iterable) / processes有余数，这将产生比进程(进程+ 1)多的任务.这有可能严重影响您的整体计算时间.在先前链接的答案中了解有关此内容的更多信息.

Otherwise, if all your taskels will need the same time to finish, you can set chunksize=len(iterable) // processes, so that tasks are only distributed once across all workers. Note that this will produce one more task than there are processes (processes + 1) in case len(iterable) / processes has a remainder. This has the potential to severely impact your overall computation time. Read more about this in the previously linked answer.

仅供参考，这是源代码的一部分，其中Pool如果未设置，则内部计算块大小:

FYI, that's the part of source code where Pool internally calculates the chunksize if not set:

    # Python 3.6, line 378 in `multiprocessing.pool.py`
    if chunksize is None:
        chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
        if extra:
            chunksize += 1
    if len(iterable) == 0:
        chunksize = 0

这篇关于与Python中的multiprocessing/pool.map无关的块大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！