在python中的不同进程之间共享列表 [英] Share a list between different processes in python

问题描述

我有以下问题.我编写了一个函数，该函数将列表作为输入，并为列表中的每个元素创建一个字典.然后，我想将此字典追加到新列表中，以便获得字典列表.我正在尝试为此生成多个进程.我在这里的问题是，我希望不同的进程访问字典列表，因为它由其他进程更新，例如，当字典列表达到一定长度时，将其打印出来. 我的示例如下:

I have the following problem. I have written a function that takes a list as input and creates a dictionary for each element in the list. I then want to append this dictionary to a new list, so I get a list of dictionaries. I am trying to spawn multiple processes for this. My problem here is that I want the different processes to access the list of dictionaries as it is updated by other processes, for example to print something once the has reached a certain length. My example would be like this:

import multiprocessing

list=['A', 'B', 'C', 'D', 'E', 'F']

def do_stuff(element):
    element_dict={}
    element_dict['name']=element
    new_list=[]
    new_list.append(element_dict)
    if len(new_list)>3:
        print 'list > 3'

###Main###
pool=multiprocessing.Pool(processes=6)
pool.map(do_stuff, list)
pool.close()

现在，我的问题是每个进程都创建自己的new_list.有没有一种方法可以在进程之间共享列表，以便所有字典都附加到同一列表中?还是在函数外部定义new_list的唯一方法?

Right now my problem is that each process creates its own new_list. Is there a way to share the list between processes, such that all dictionaries are appended to the same list? Or is the only way to define the new_list outside of the function?

推荐答案

一种方法是使用管理器对象并从中创建共享列表对象:

One way is to use a manager object and create your shared list object from it:

from multiprocessing import Manager, Pool

input_list = ['A', 'B', 'C', 'D', 'E', 'F']

manager = Manager()
shared_list = manager.list()

def do_stuff(element):
    global shared_list
    element_dict = {}
    element_dict['name'] = element
    shared_list.append(element_dict)
    if len(shared_list) > 3:
        print('list > 3')

pool = Pool(processes=6)
pool.map(do_stuff, input_list)
pool.close()

请记住，与线程不同，进程不共享内存空间. (生成时，每个进程都会获得其生成进程的内存足迹的副本，然后与之一起运行.)因此，它们只能通过某种形式的IPC进行通信(进程间通信).在Python中，这样的方法之一是multiprocessing.Manager及其公开的数据结构，例如list或dict.这些代码在代码中的用法与它们的内置等效项一样容易，但是在后台使用了某种形式的IPC(可能是套接字).

Remember, unlike threads, processes do not share memory space. (When spawned, each process gets its own copy of the memory footprint of the spawning process, and then runs with it.) So they can only communicate via some form of IPC (interprocess communication). In Python, one such method is multiprocessing.Manager and the data structures it exposes, e.g. list or dict. These are used in code as easily as their built-in equivalents, but under the hood utilize some form of IPC (sockets probably).

这篇关于在python中的不同进程之间共享列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！