Python:Multiprocessing Map需要更长的时间才能完成最后几个过程 [英] Python: Multiprocessing Map takes longer to complete last few processes

问题描述

在Python中，我试图运行150-200个进程.我将这150样东西放在一个数组中，并将这个数组拆分成每个10个元素的多个数组.

In Python, I'm trying to run 150-200 processes. I have these 150 things in an array, and I've split this array up into multiple arrays of 10 elements each.

现在，我运行一个Multiprocessing Map，一次包含10个元素.完成所有10个步骤后，我们进入下一个10个，依此类推.

Now, I run a Multiprocessing Map, with 10 elements at a time. Once all 10 are complete, we go onto the next 10, and so on.

现在，问题是:第九和第十个进程几乎总是比其余进程慢.有什么理由吗?我不是最有效的方法吗?

Now, the problem: The ninth and tenth process are almost ALWAYS slower than the rest. Is there a reason for that? Am I not doing this the most efficient way?

**我将无法共享此代码.那么，您对为什么会发生这种情况有任何想法吗?

** I won't be able to share the code for this. So do you have any ideas as to why this may be happening?

谢谢.

推荐答案

这是由于Pool.map在工作人员之间分发数据的方式所致.使用chunksize=1作为参数，即map(...,..., chunksize=1).

This is due to the way Pool.map distributes the data among the workers. Use chunksize=1 as a parameter, i.e. map(...,..., chunksize=1).

类似的问题在以下解释: python多处理映射对最后一个进程的处理不当

A similar problem is explained in: python multiprocessing map mishandling of last processes

这篇关于Python:Multiprocessing Map需要更长的时间才能完成最后几个过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！