通过multiprocessing.pool.map传递kwarg [英] passing kwargs with multiprocessing.pool.map
问题描述
我想通过Pool.map()将关键字参数传递给我的worker函数.在搜索论坛时,我找不到明确的示例.
I would like to pass keyword arguments to my worker-function with Pool.map(). I can't find a clear example of this when searching forums.
示例代码:
import multiprocessing as mp
def worker((x,y), **kwargs):
kwarg_test = kwargs.get('kwarg_test', False)
print("kwarg_test = {}".format(kwarg_test))
if kwarg_test:
print("Success")
return x*y
def wrapper_process(**kwargs):
jobs = []
pool=mp.Pool(4)
for i, n in enumerate(range(4)):
jobs.append((n,i))
pool.map(worker, jobs) #works
pool.map(worker, jobs, kwargs) #how to do this?
def main(**kwargs):
worker((1,2),kwarg_test=True) #accepts kwargs
wrapper_process(kwarg_test=True)
if __name__ == "__main__":
main()
输出:
kwarg_test = True
Success
kwarg_test = False
kwarg_test = False
kwarg_test = False
kwarg_test = False
TypeError: unsupported operand type(s) for //: 'int' and 'dict'
类型错误与在multiprocessing.Pool或Queue内部解析参数有关,我还尝试了其他几种语法,例如列出kwargs. [kwargs,kwargs,kwargs,kwargs],以及多次尝试将kwarg列入工作清单,但没有运气.我在multiprocessing.pool中从map到map_async跟踪了代码,并得到了
task_batches = Pool._get_tasks(func, iterable, chunksize)
当我遇到生成器结构时,在pool.py中.我很高兴将来能进一步了解这一点,但现在我只是想找出答案:
The type error has to do with parsing arguments inside of multiprocessing.Pool or Queue, and I have tried several other syntaxes, like making a list of the kwargs; [kwargs, kwargs, kwargs, kwargs], as well as several attempts to include the kwarg in the jobs list but no luck. I traced the code in multiprocessing.pool from map to map_async and got as far as
task_batches = Pool._get_tasks(func, iterable, chunksize)
in pool.py when I encountered the generator structure. I'm happy to learn more about this in future but for now I am just trying to find out:
是否有一种简单的语法可以允许kwargs通过pool.map传递?
Is there a simple syntax for allowing the passing of kwargs with pool.map?
推荐答案
如果要遍历其他参数,请使用@ArcturusB的答案.
If you want to iterate over the other arguments, use @ArcturusB's answer.
如果您只想传递它们,且每次迭代具有相同的值,则可以执行以下操作:
If you just want to pass them, having the same value for each iteration, then you can do this:
from functools import partial
pool.map(partial(worker, **kwargs), jobs)
Partial 将参数绑定到函数.旧版本的Python 无法序列化部分对象.
Partial 'binds' arguments to a function. Old versions of Python cannot serialize partial objects though.
这篇关于通过multiprocessing.pool.map传递kwarg的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!