Python 3.6+:嵌套多处理管理器导致 FileNotFoundError [英] Python 3.6+: Nested multiprocessing managers cause FileNotFoundError

问题描述

所以我试图在一个 dicts 的 dicts 上使用 multiprocessing Manager，这是我最初的尝试:

So I'm trying to use multiprocessing Manager on a dict of dicts, this was my initial try:

from multiprocessing import Process, Manager

def task(stat):
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':
    test = Manager().dict({'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 0, 'Y1': 0}, 'z': 0})

    p = Process(target=task, args=(test,))
    p.start()
    p.join()

    print(test)

当然，当我运行它时，输出不是我所期望的，z 正确更新而 y 不变！这是输出:

of course when I run this, the output is not what I expect, z updates correctly while y is unchanged! This is the output:

{'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 0, 'Y1': 0}, 'z': 1}

然后我用谷歌搜索，并在这里找到了一个解释，显然嵌套的 dicts 也必须是 Manager().dict()s 而不是普通的 python dicts(自 Python 3.6 起可能).所以我做了以下事情:

Then I googled, and found an explanation here, apparently the nested dicts have to also be Manager().dict()s rather than normal python dicts (possible since Python 3.6). So I did the following:

from multiprocessing import Process, Manager

def task(stat):
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':
    test = Manager().dict({'x': Manager().dict({'X0': 10, 'X1': 20}), 'y': Manager().dict({'Y0': 0, 'Y1': 0}), 'z': 0})

    p = Process(target=task, args=(test,))
    p.start()
    p.join()

    print(test)
    print(test['y'])

但它并没有正常工作，而是出现了这个无法解释的错误，为了清楚起见，它分为三个部分.第一部分对应于 test['y']['Y0'] += 5 而第二部分是简单的 print(test)，最后一部分是输出print(test['y'])

But instead of it working properly, I get this unexplainable error(s), split to three parts for clarity. The first part corresponds to the test['y']['Y0'] += 5 while the second is simply the print(test) and the last is the output of print(test['y'])

Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "shit.py", line 5, in task
    test['y']['Y0'] += 5
  File "<string>", line 2, in __getitem__
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 920, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 770, in __init__
    self._incref()
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 824, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory




{'x': <DictProxy object, typeid 'dict' at 0x7f01de2c5860>, 'y': <DictProxy object, typeid 'dict' at 0x7f01de2c5898>, 'z': 1}




Traceback (most recent call last):
  File "test.py", line 16, in <module>
    print(test['y'])
  File "<string>", line 2, in __getitem__
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 920, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 770, in __init__
    self._incref()
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 824, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

我不确定为什么会发生这种情况.内部字典显然已经创建(如输出的第二部分所示).但是由于某种原因，它们根本无法读取或写入！.为什么会发生这种情况?

I'm not sure why this happens. The inner dicts evidently get created (as shown by the second part of the output). But for some reason, they cannot be read or written at all!. Why does this happen?

额外:如果我通过 python 控制台(而不是脚本)运行相同的 python 代码，错误将从 FileNotFoundError 更改为 ConnectionRefusedError.但具有相同的精确追溯！

Extra: If I run the same python code through a python console (rather than a script) the error changes from FileNotFoundError to ConnectionRefusedError. But with the same exact traceback!

推荐答案

在 Manager().dict() 中使用 Manager() 你正在开始一个新的经理 -每次都处理，所以你真的是嵌套管理器(就像标题所说的那样)，这不是它应该的方式.相反，您需要做的是实例化 one Manager，然后在该 Manager 实例上创建字典:

With Manager() in Manager().dict() you are starting a new manager-process each time, so you are really nesting managers (like the title says) and this is not the way it's supposed to be. What you need to do instead, is instantiate one Manager and then create dictionaries on that manager instance:

from multiprocessing import Process, Manager
from multiprocessing.managers import DictProxy


def task(test):  # use parameter `test`, else you rely on forking
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':

    with Manager() as m:

        test = m.dict({'x': m.dict({'X0': 10, 'X1': 20}),
                       'y': m.dict({'Y0': 0, 'Y1': 0}),
                       'z': 0})

        p = Process(target=task, args=(test,))
        p.start()
        p.join()

        print(test)
        print(test['y'])

        # convert to normal dict before closing manager for persistence
        # in parent or for printing dict behind proxies
        d = {k: dict(v) if isinstance(v, DictProxy) else v
             for k, v in test.items()}

    print(d) # Manager already closed here

示例输出:

{'x': <DictProxy object, typeid 'dict' at 0x7f98cdaaa588>, 'y': <DictProxy object, typeid 'dict' at 0x7f98cda99c50>, 'z': 1}
{'Y0': 5, 'Y1': 0}
{'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 5, 'Y1': 0}, 'z': 1}

Process finished with exit code 0

如果您计划从多个进程修改管理器对象，您还需要使用 Manager.Lock.

You would also need to use Manager.Lock in case you plan to modify manager-objects from multiple processes.

这篇关于Python 3.6+:嵌套多处理管理器导致 FileNotFoundError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！