Python多重处理-将字典列表传递给池 [英] Python multiprocessing - Passing a list of dicts to a pool

问题描述

这个问题可能是重复的.但是，我在这个话题上读了很多东西，但没有找到与我的情况相符的东西-或至少我不理解它.

This question may be a duplicate. However, I read lot of stuff around on this topic, and I didn't find one that matches my case - or at least, I didn't understood it.

很抱歉给您带来麻烦.

我想做的是相当普遍的，将kwargs列表传递给pool.starmap()，以实现多处理.

What I'm trying to do is fairly common, passing a list of kwargs to pool.starmap(), to achieve multiprocessing.

这是我减轻的情况:

def compute(firstArg, **kwargs): # A function that does things
    # Fancy computing stuff...
    # Saving to disk...
    return True

if __name__ ==  '__main__':
    args = [{
        'firstArg': "some data",
        'otherArg': "some other data"
        'andAnotherArg': x*2
    } for x in range(42)]

    pool = Pool(4)
    pool.starmap(compute, args)
    pool.close()
    pool.terminate()

我认为starmap()将解压缩字典并将其作为关键字args传递给compute()，但是要查看

I supposed starmap() will unpack the dict and pass it to compute() as keyword args, but looking at the source code (see also l.46), it sends only keys (or values ?).

所以它引起了:

TypeError: compute() takes 1 positional argument but 3 were given

这必须是一种清晰，直接的方法...任何帮助将不胜感激.

It must be a clear, straight forward way to do this... Any help would be appreciated.

这是一个非常类似的问题: Python Multiprocessing-如何传递kwarg起作用?

Here's a quite similar question : Python Multiprocessing - How to pass kwargs to function?

推荐答案

您可以使用一个小型代理功能:

You could use a tiny proxy function:

def proxy(Dict):
    return compute(**Dict)

pool.map(proxy, args)

或者，因为不需要proxy函数污染名称空间:

Or, since you don't need the proxy function polluting the namespace:

pool.map(lambda Dict: compute(**Dict), args)

这篇关于Python多重处理-将字典列表传递给池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！