如何结合多处理和事件 [英] How to combine multiprocessing and eventlet

问题描述

我有一个任务需要启动2个进程，并且每个进程中需要启动2个线程才能真正起作用. 下面是我用来模拟用例的源代码.

I have a task need to start 2 processes and within each process need to start 2 threads to do really work. Below is the source code I used to simulate my use case.

import multiprocessing
import eventlet

def subworker(num1, num2):
    print 'Start subworker %d,%d\n' % (num1, num2)
    eventlet.sleep(10)
    print 'End subworker %d,%d\n' % (num1, num2)

def worker(**kwargs):
    number = kwargs['number']
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(subworker, number, 1)
    pool.spawn_n(subworker, number, 2)
    pool.waitall()

def launcher(number):
    kwargs = {'number': number}
    th = multiprocessing.Process(target=worker, kwargs=kwargs)
    th.start()
    while True:
        if not th.is_alive():
            break
        eventlet.sleep(0)

    th.join()


def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    pool.spawn_n(launcher, 2)
    pool.waitall()

main()

当我运行此python脚本时，我的预期输出如下:

When I run this python script my expected output is something like:

Start subworker 1,1 
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2

但是我真正得到的是:

Start subworker 1,1
Traceback (most recent call last):

  File "/Users/leehom/python_local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Start subworker 1,2
    func(*args, **kwargs)

  File "/Users/leehom/Desktop/home/work_dir/source/snips/Test_multiprocessing_and_eventlet.py", line 27, in launcher
Start subworker 2,1
    if not th.is_alive():

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 155, in is_alive
Start subworker 2,2
    assert self._parent_pid == os.getpid(), 'can only test a child process'

AssertionError: can only test a child process
Start subworker 1,1
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
Process finished with exit code 0

似乎启动器被两次调用.我不明白为什么会这样.

Seems launcher was called twice. I do not understand why this happen.

如果我在主函数中注释了一行

If I comment one line in my main function

def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    # pool.spawn_n(launcher, 2)
    pool.waitall()

结果正确:

Start subworker 1,1
Start subworker 1,2
End subworker 1,1
End subworker 1,2

任何人都知道我该如何解决此问题以及为什么会发生此问题?

Anyone knows how I can fix this and why this problem happen?

推荐答案

截至2018年1月，Eventlet和多处理不能很好地协同工作.最好的选择是从外部生成工作进程.第二好的选择是os.fork()创建工作进程，然后创建工作进程 import eventlet.

As of 2018-01, Eventlet and multiprocessing don't work well together. Your best option is to spawn worker processes externally. Second best option is os.fork() to create worker processes and only then import eventlet.

订阅此问题，以在解决多处理兼容性时得到通知. https://github.com/eventlet/eventlet/issues/147

Subscribe to this issue to be notified when multiprocessing compatibility is resolved. https://github.com/eventlet/eventlet/issues/147

这篇关于如何结合多处理和事件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！