带有maxtasksperchild的池产生相等的PID [英] multiprocessing.Pool with maxtasksperchild produces equal PIDs
问题描述
我需要在一个与所有其他内存完全隔离的进程中运行一个函数几次.我想为此使用multiprocessing
(因为我需要序列化来自函数的复杂输出).我将start_method
设置为'spawn'
并使用带有maxtasksperchild=1
的池.我希望每个任务都有不同的过程,因此会看到不同的PID:
I need to run a function in a process, which is completely isolated from all other memory, several times. I would like to use multiprocessing
for that (since I need to serialize a complex output coming from the functions). I set the start_method
to 'spawn'
and use a pool with maxtasksperchild=1
. I would expect to get a different process for each task, and therefore see a different PID:
import multiprocessing
import time
import os
def f(x):
print("PID: %d" % os.getpid())
time.sleep(x)
complex_obj = 5 #more complex axtually
return complex_obj
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
pool = multiprocessing.Pool(4, maxtasksperchild=1)
pool.map(f, [5]*30)
pool.close()
但是我得到的输出是:
$ python untitled1.py
PID: 30010
PID: 30009
PID: 30012
PID: 30011
PID: 30010
PID: 30009
PID: 30012
PID: 30011
PID: 30018
PID: 30017
PID: 30019
PID: 30020
PID: 30018
PID: 30019
PID: 30017
PID: 30020
...
因此,不会在每个任务之后重新生成进程.每次都有一种自动获取新PID的自动方法(即,无需为每个进程集启动新池)吗?
So the processes are not being respawned after every task. Is there an automatic way of getting a new PID each time (ie without starting a new pool for each set of processes)?
推荐答案
您还需要在对pool.map
的调用中指定chunksize=1
.否则,根据对工作进程的理解,将可迭代项中的多个项目捆绑在一起成为一个任务":
You need to also specify chunksize=1
in the call to pool.map
. Otherwise, multiple items in your iterable get bundled together into one "task" from the perception of the worker processes:
import multiprocessing
import time
import os
def f(x):
print("PID: %d" % os.getpid())
time.sleep(x)
complex_obj = 5 #more complex axtually
return complex_obj
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
pool = multiprocessing.Pool(4, maxtasksperchild=1)
pool.map(f, [5]*30, chunksize=1)
pool.close()
输出现在没有重复的PID:
Output doesn't have repeated PIDs now:
PID: 4912
PID: 4913
PID: 4914
PID: 4915
PID: 4938
PID: 4937
PID: 4940
PID: 4939
PID: 4966
PID: 4965
PID: 4970
PID: 4971
PID: 4991
PID: 4990
PID: 4992
PID: 4993
PID: 5013
PID: 5014
PID: 5012
这篇关于带有maxtasksperchild的池产生相等的PID的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!