异步 + 多处理 + Unix [英] asyncio + multiprocessing + unix

问题描述

我有一个具有以下逻辑的宠物项目:

I have a pet project with the following logic:

import asyncio, multiprocessing

async def sub_main():
    print('Hello from subprocess')

def sub_loop():
    asyncio.get_event_loop().run_until_complete(sub_main())

def start():
    multiprocessing.Process(target=sub_loop).start()

start()

如果你运行它，你会看到:

If you run it, you'll see:

Hello from subprocess

那很好.但我要做的是让 start() 协程代替:

That is good. But what I have to do is to make start() coroutine instead:

async def start():
    multiprocessing.Process(target=sub_loop).start()

要运行它，我必须这样做:

To run it, I have to do something like that:

asyncio.get_event_loop().run_until_complete(start())

问题是:当创建子进程时，它会克隆整个 Python 环境，因此事件循环已经在那里运行:

Here is the issue: when sub process is created, it gets the whole Python environment cloned, so event loop is already running there:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test.py", line 7, in sub_loop
    asyncio.get_event_loop().run_until_complete(sub_main())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 361, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 326, in run_forever
    raise RuntimeError('Event loop is running.')
RuntimeError: Event loop is running.

我试图在没有运气的情况下在子进程端销毁它，但我认为正确的方法是防止它与子进程共享.有可能吗?

I tried to destroy it on subprocess side with no luck but I think that the correct way is to prevent its sharing with subprocess though. Is it possible somehow?

更新:这是完整的失败代码:

UPDATE: Here is the full failing code:

import asyncio, multiprocessing

import asyncio.unix_events

async def sub_main():
    print('Hello from subprocess')

def sub_loop():
    asyncio.get_event_loop().run_until_complete(sub_main())


async def start():
    multiprocessing.Process(target=sub_loop).start()

asyncio.get_event_loop().run_until_complete(start())

推荐答案

首先，您应该考虑使用 loop.run_in_executor 带有 ProcessPoolExecutor 如果您打算从循环内运行 python 子进程.至于你的问题，你可以使用 事件循环策略用于设置新循环:

First, you should consider using loop.run_in_executor with a ProcessPoolExecutor if you plan to run python subprocesses from within the loop. As for your problem, you can use the event loop policy functions to set a new loop:

import asyncio
from concurrent.futures import ProcessPoolExecutor

async def sub_main():
    print('Hello from subprocess')

def sub_loop():
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(sub_main())

async def start(executor):
    await asyncio.get_event_loop().run_in_executor(executor, sub_loop)

if __name__ == '__main__':
    executor = ProcessPoolExecutor()
    asyncio.get_event_loop().run_until_complete(start(executor))

这篇关于异步 + 多处理 + Unix的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！