multiprocessing.Manager嵌套的共享对象不适用于Queue [英] multiprocessing.Manager nested shared objects doesn't work with Queue

问题描述

Python文档:

在3.6版中已更改:共享对象能够嵌套.例如，诸如共享列表之类的共享容器对象可以包含其他共享对象，所有这些共享对象都将由SyncManager进行管理和同步.

Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

这确实适用于list和dict.但是，如果我尝试在共享dict中创建共享Queue，则会收到错误消息:

This does work with list and dict. However, if I try to create a shared Queue inside a shared dict, I get an error:

>>> from multiprocessing import Manager
>>> m = Manager()
>>> d = m.dict()
>>> d['a'] = m.list()
>>> d['b'] = m.dict()
>>> d['c'] = m.Queue()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2, in __setitem__
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
---------------------------------------------------------------------------

类似于 https://hg.python.org/cpython/rev/39e7307f9aee是引入嵌套共享对象的变更集.

Seems like https://hg.python.org/cpython/rev/39e7307f9aee is the changeset which introduced nested shared objects.

推荐答案

该错误是由AutoProxy当前未处理所有BaseProxy自变量引起的.有一个尚未合并的拉请求.您要么需要猴子补丁AutoProxy，要么调查multiprocessing.managers.py并在补丁此处直接访问您的源代码.

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

修复补丁中的两行以防止服务器进程中的内存泄漏非常重要. manager_owned标志用于通知BaseProxy代码，何时跳过管理者拥有的代理的引用增量(通过嵌套).

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).

这篇关于multiprocessing.Manager嵌套的共享对象不适用于Queue的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！