python多处理锁定问题 [英] python multiprocessing lock issue
问题描述
我想与python多处理模块一起添加字典列表.
I want to add a list of dicts together with python multiprocessing module.
这是我的代码的简化版:
Here is a simplified version of my code:
#!/usr/bin/python2.7
# -*- coding: utf-8 -*-
import multiprocessing
import functools
import time
def merge(lock, d1, d2):
time.sleep(5) # some time consuming stuffs
with lock:
for key in d2.keys():
if d1.has_key(key):
d1[key] += d2[key]
else:
d1[key] = d2[key]
l = [{ x % 10 : x } for x in range(10000)]
lock = multiprocessing.Lock()
d = multiprocessing.Manager().dict()
partial_merge = functools.partial(merge, d1 = d, lock = lock)
pool_size = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes = pool_size)
pool.map(partial_merge, l)
pool.close()
pool.join()
print d
-
运行此脚本时出现此错误.我该如何解决?
I get this error when running this script. How shall I resolve this?
RuntimeError: Lock objects should only be shared between processes through inheritance
在这种情况下是否需要merge
函数中的lock
?还是python会照顾好它?
is the lock
in merge
function needed in this condition? or python will take care of it?
我认为map
应该做的是将某物从一个列表映射到另一列表,而不是将一个列表中的所有内容转储到单个对象.那么,有没有更优雅的方式来做这些事情呢?
I think what's map
supposed to do is to map something from one list to another list, not dump all things in one list to a single object. So is there a more elegant way to do such things?
推荐答案
以下代码应在Python 2和3中跨平台运行(即在Windows上也是如此).它使用进程池初始化程序将管理器字典设置为每个子进程中的全局变量.
The following should run cross-platform (i.e. on Windows, too) in both Python 2 and 3. It uses a process pool initializer to set the manager dict as a global in each child process.
仅供参考:
- Using a lock is unnecessary with a manager dict.
- The number of processes in a
Pool
defaults to the CPU count. - If you're not interested in the result, you can use
apply_async
instead ofmap
.
import multiprocessing
import time
def merge(d2):
time.sleep(1) # some time consuming stuffs
for key in d2.keys():
if key in d1:
d1[key] += d2[key]
else:
d1[key] = d2[key]
def init(d):
global d1
d1 = d
if __name__ == '__main__':
d1 = multiprocessing.Manager().dict()
pool = multiprocessing.Pool(initializer=init, initargs=(d1, ))
l = [{ x % 5 : x } for x in range(10)]
for item in l:
pool.apply_async(merge, (item,))
pool.close()
pool.join()
print(l)
print(d1)
这篇关于python多处理锁定问题的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!