Gevent Monkeypatching破坏多处理 [英] Gevent monkeypatching breaking multiprocessing

问题描述

我正在尝试使用多重处理池来运行一组进程，每个进程都将运行一个greenlet的gevent池.原因是网络活动很多，CPU活动很多，因此要最大化带宽和所有CPU核心，我需要多个进程以及gevent的异步猴子补丁程序.我正在使用multiprocessing的管理器来创建一个队列，进程将访问该队列以获取要处理的数据.

I am attempting to use multiprocessing's pool to run a group of processes, each of which will run a gevent pool of greenlets. The reason for this is that there is a lot of network activity, but also a lot of CPU activity, so to maximise my bandwidth and all of my CPU cores, I need multiple processes AND gevent's async monkey patching. I am using multiprocessing's manager to create a queue which the processes will access to get data to process.

这是代码的简化片段:

import multiprocessing

from gevent import monkey
monkey.patch_all(thread=False)

manager = multiprocessing.Manager()
q = manager.Queue()

这是它产生的异常:

Traceback (most recent call last):
  File "multimonkeytest.py", line 7, in <module>
    q = manager.Queue()
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 667, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 565, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 175, in Client
    answer_challenge(c, authkey)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 409, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
 IOError: [Errno 35] Resource temporarily unavailable

我认为这一定是由于普通套接字模块和gevent套接字模块的行为有所不同.

I believe this must be due to some difference between the behaviour of the normal socket module and gevent's socket module.

如果我在子流程中进行了monkeypatch，则成功创建了队列，但是当子流程尝试从队列中获取get()时，会发生非常相似的异常.由于在子进程中执行了大量网络请求，因此确实需要对套接字进行修补.

If I monkeypatch within the subprocess, The queue is created successfully, but when the subprocess tries to get() from the queue, a very similar exception occurs. The socket does need to be monkeypatched due to doing large numbers of network requests in the subprocesses.

我的gevent版本是最新版本:

My version of gevent, which I believe is the latest:

>>> gevent.version_info
(1, 0, 0, 'alpha', 3)

有什么想法吗?

推荐答案

使用monkey.patch_all(thread=False, socket=False)

在相似的情况下，我遇到了相同的问题，并在patch_socket()函数下通过_socket.socket = socket.socket将此问题追溯到了gevent/monkey.py中的第115行.注释掉该行可防止损坏.

I have run into the same issue in a similar situation and tracked this down to line 115 in gevent/monkey.py under the patch_socket() function: _socket.socket = socket.socket. Commenting this line out prevents the breakage.

这是gevent用其自身替换stdlib socket库的地方. multiprocessing.connection广泛使用socket库，并且显然不能容忍此更改.

This is where gevent replaces the stdlib socket library with its own. multiprocessing.connection uses the socket library quite extensively, and is apparently not tolerant to this change.

特别是，在导入的模块执行gevent.monkey.patch_all()调用而无需设置socket=False的任何情况下，您都会看到此消息.就我而言，是grequests做到了这一点，我必须重写套接字模块的修补程序才能解决此错误.

Specifically, you will see this in any scenario where a module you import performs a gevent.monkey.patch_all() call without setting socket=False. In my case it was grequests that did this, and I had to override the patching of the socket module to fix this error.

这篇关于Gevent Monkeypatching破坏多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！