无法将项目追加到多处理共享列表 [英] Cannot append items to multiprocessing shared list

问题描述

我正在使用多处理为我的应用程序创建子流程. 我还在流程和子流程之间共享一个字典.

I'm using multiprocessing to create sub-process to my application. I also share a dictionary between the process and the sub-process.

我的代码示例:

主要过程:

from multiprocessing import Process, Manager
manager = Manager()
shared_dict = manager.dict()
p = Process(target=mysubprocess, args=(shared_dict,))
p.start()
p.join()
print shared_dict

我的子流程:

def mysubprocess(shared_dict):
  shared_dict['list_item'] = list()
  shared_dict['list_item'].append('test')
  print shared_dict

在两种情况下，打印值均为: {'list_item': []}

In both cases the printed value is : {'list_item': []}

可能是什么问题? 谢谢

What could be the problem? Thanks

推荐答案

Manager.dict将为您提供一个dict，在其中直接更改将在进程之间传播，但是它不会检测您是否更改了dict中包含的对象(就像存储在"list_item"下的列表一样).请参见 SyncManager文档

Manager.dict will give you a dict where direct changes will be propagated between the processes, but it doesn't detect if you change objects contained in the dict (like the list stored under "list_item"). See the note at the bottom of the SyncManager documentation:

注意:对dict和list代理中的可变值或项的修改不会通过管理器传播，因为代理无法知道何时修改其值或项.要修改此类项目，您可以将修改后的对象重新分配给容器代理.

Note: Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy.

因此，在您的示例中，当您在dict中设置列表时，该列表已同步，但是追加不会触发其他同步.

So in your example the list gets synced when you set it in the dict, but the append doesn't trigger another sync.

您可以通过在dict中重新分配密钥来解决此问题:

You can get around that by re-assigning the key in the dict:

from multiprocessing import Process, Manager

def mysubprocess(shared_dict):
    item = shared_dict['list_item'] = list()
    item.append('test')
    shared_dict['list_item'] = item
    print 'subprocess:', shared_dict

manager = Manager()
shared_dict = manager.dict()
p = Process(target=mysubprocess, args=(shared_dict,))
p.start()
p.join()
print 'main process:', shared_dict

但是，如果列表长得太长，可能会变得效率低下-整个列表将被序列化，并发送给每个追加的管理器进程.在这种情况下，一种更好的方法是直接使用SyncManager.list创建共享列表(尽管如果列表中的元素可变，您仍然会遇到相同的问题-您需要在列表中重置它们，以便在列表之间发送它们进程).

But that might get inefficient if the list is going to grow long - the whole list will be serialised and sent to the manager process for each append. A better way in that case would be to make the shared list directly with SyncManager.list (although you'll still have the same problem if the elements of the list are mutable - you need to reset them in the list to send them between the processes).

这篇关于无法将项目追加到多处理共享列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！