使用Python 2.7/Windows运行多处理作业 [英] Run a multiprocessing job with Python 2.7 / Windows
问题描述
基于此答案,我想在Python 2.7/Windows上运行此multiprocessing
作业:
Based on this answer, I'd like to run this multiprocessing
job with Python 2.7 / Windows:
def main():
import itertools as it
from multiprocessing import Pool
def dothejob(i, j, k):
print i, j, k
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
def jobWrapper(args):
return dothejob(*args)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
main()
main()
和最后两行是必需的,因为如果没有它们,则存在一个众所周知的错误:
The main()
and the last two lines are necessary because without them, there's the well known bug:
这可能意味着您在Windows上并且拥有 忘记在主模块中使用适当的习惯用法:
This probably means that you are on Windows and you have forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
....
但是即使如此,我仍然收到此错误:
But even with that, I get this error:
文件"C:\ Users \ User \ Desktop \ test.py",主行中的第14行 res = pool.map(jobWrapper,the_args) 地图中的文件"C:\ Python27 \ lib \ multiprocessing \ pool.py",第251行 返回self.map_async(func,可迭代,块大小).get() 在get中的文件"C:\ Python27 \ lib \ multiprocessing \ pool.py",行558 提高自我价值 cPickle.PicklingError:无法腌制:属性查找> 内置.功能失败
File "C:\Users\User\Desktop\test.py", line 14, in main res = pool.map(jobWrapper, the_args) File "C:\Python27\lib\multiprocessing\pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "C:\Python27\lib\multiprocessing\pool.py", line 558, in get raise self._value cPickle.PicklingError: Can't pickle : attribute lookup >builtin.function failed
此处涉及cPickle
的位置,以及此错误的原因/解决方法?
Where is there a cPickle
involved here and why this error / how to solve it?
推荐答案
所有定义都必须在模块范围内:
All definitions must be on module scope:
import itertools as it
from multiprocessing import Pool, freeze_support
def dothejob(i, j, k):
print i, j, k
def jobWrapper(args):
return dothejob(*args)
def main():
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
freeze_support() #you need this in windows
main()
您还需要在Windows中进行freeze_support
调用
You also need freeze_support
call in windows
这篇关于使用Python 2.7/Windows运行多处理作业的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!