Python在并行进程之间共享字典 [英] Python sharing a dictionary between parallel processes

问题描述

我想在进程之间共享一个字典，如下所示:

I want to share a dictionary between my processes as follows:

def f(y,x):
    y[x]=[x*x]                                                          

if __name__ == '__main__':
    pool = Pool(processes=4)
    inputs = range(10)
    y={}                             
    result = pool.map(f,y,inputs)

y返回{}.我该如何运作?

The y returns {}. How can I make it work?

谢谢

推荐答案

这看起来像您正在使用multiprocessing模块.您没有说，这是重要的信息.

This looks like you are using the multiprocessing module. You didn't say, and that's an important bit of information.

multiprocessing.Pool()实例上的.map()函数带有两个参数:函数和序列.将使用序列中的连续值调用该函数.您正在尝试传递y和一个序列，但这将不起作用.

The .map() function on a multiprocessing.Pool() instance takes two arguments: a function, and a sequence. The function will be called with successive values from the sequence. You are trying to pass y and a sequence and that won't work.

您可以这样编写一个序列:((y, x) for x in input)，并编写一个函数，该函数接受一个参数，一个元组，然后对其执行正确的操作.

You could make one sequence like this: ((y, x) for x in input), and write a function that takes one argument, a tuple, and then does the right thing with it.

我仅使用multiprocessing.Pool()运行读取和写入磁盘文件的进程，因此我不确定基本思想在这里如何工作:我不确定您是否能够像尝试尝试的那样变异dict做.

I have only used multiprocessing.Pool() to run processes that read and write disk files, so I am not sure how the basic idea will work here: I am not sure you will be able to mutate a dict like you are trying to do.

我建议只从函数中传回值，然后收集它们以得出最终结果.我刚刚测试了这段代码，它的工作原理是:

I'd recommend just passing values back from your function, then collecting them to make your final result. I just tested this code and it works:

import multiprocessing as mp

def f(x):
    return (x, x*x)

if __name__ == '__main__':
    pool = mp.Pool()
    inputs = range(10)
    result = dict(pool.map(f, inputs))

result设置为:{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

让我们对其进行更改，以便代替计算x*x它将把x提高到一定的功率，并将提供该功率.让我们接受一个字符串键参数.这意味着f()需要接受一个元组参数，其中元组将为(key, x, p)并且将计算x**p.

Let's change it so that instead of computing x*x it will raise x to some power, and the power will be provided. And let's make it take a string key argument. This means that f() needs to take a tuple argument, where the tuple will be (key, x, p) and it will compute x**p.

import multiprocessing as mp

def f(tup):
    key, x, p = tup  # unpack tuple into variables
    return (key, x**p)

if __name__ == '__main__':
    pool = mp.Pool()
    inputs = range(10)
    inputs = [("1**1", 1, 1), ("2**2", 2, 2), ("2**3", 2, 3), ("3**3", 3, 3)]
    result = dict(pool.map(f, inputs))

如果您有多个序列，并且需要将它们连接在一起以构成上述单个序列，请考虑使用zip()或itertools.product.

If you have several sequences and you need to join them together to make a single sequence for the above, look into using zip() or perhaps itertools.product.

这篇关于Python在并行进程之间共享字典的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！