如何使用 python 的 asyncio 模块正确创建和运行并发任务? [英] How to properly create and run concurrent tasks using python's asyncio module?

问题描述

我试图正确理解和实现两个同时运行的Task 对象使用 Python 3 的相对较新的 asyncio 模块.

I am trying to properly understand and implement two concurrently running Task objects using Python 3's relatively new asyncio module.

简而言之，asyncio 似乎旨在通过事件循环处理异步进程和并发Task 执行.它提倡使用 await(应用于异步函数)作为等待和使用结果的无回调方式，而不会阻塞事件循环.(期货和回调仍然是一个可行的选择.)

In a nutshell, asyncio seems designed to handle asynchronous processes and concurrent Task execution over an event loop. It promotes the use of await (applied in async functions) as a callback-free way to wait for and use a result, without blocking the event loop. (Futures and callbacks are still a viable alternative.)

它还提供了 asyncio.Task() 类，它是 Future 的专门子类，旨在包装协程.最好使用 asyncio.ensure_future() 方法调用.异步任务的预期用途是允许独立运行的任务与同一事件循环中的其他任务同时"运行.我的理解是 Tasks 连接到事件循环，然后它会自动在 await 语句之间驱动协同程序.

It also provides the asyncio.Task() class, a specialized subclass of Future designed to wrap coroutines. Preferably invoked by using the asyncio.ensure_future() method. The intended use of asyncio tasks is to allow independently running tasks to run 'concurrently' with other tasks within the same event loop. My understanding is that Tasks are connected to the event loop which then automatically keeps driving the coroutine between await statements.

我喜欢无需使用 Executor 类，但我没有找到太多关于实现的详细说明.

I like the idea of being able to use concurrent Tasks without needing to use one of the Executor classes, but I haven't found much elaboration on implementation.

这就是我目前的做法:

import asyncio

print('running async test')

async def say_boo():
    i = 0
    while True:
        await asyncio.sleep(0)
        print('...boo {0}'.format(i))
        i += 1

async def say_baa():
    i = 0
    while True:
        await asyncio.sleep(0)
        print('...baa {0}'.format(i))
        i += 1

# wrap in Task object
# -> automatically attaches to event loop and executes
boo = asyncio.ensure_future(say_boo())
baa = asyncio.ensure_future(say_baa())

loop = asyncio.get_event_loop()
loop.run_forever()

在尝试同时运行两个循环任务的情况下，我注意到除非任务具有内部 await 表达式，否则它会卡在 while循环，有效地阻止其他任务运行(很像普通的 while 循环).但是，一旦任务必须 (a) 等待，它们似乎可以同时运行而没有问题.

In the case of trying to concurrently run two looping Tasks, I've noticed that unless the Task has an internal await expression, it will get stuck in the while loop, effectively blocking other tasks from running (much like a normal while loop). However, as soon the Tasks have to (a)wait, they seem to run concurrently without an issue.

因此，await 语句似乎为事件循环提供了在任务之间来回切换的立足点，从而产生并发效果.

Thus, the await statements seem to provide the event loop with a foothold for switching back and forth between the tasks, giving the effect of concurrency.

带有内部 await 的示例输出:

Example output with internal await:

running async test
...boo 0
...baa 0
...boo 1
...baa 1
...boo 2
...baa 2

示例输出没有内部await:

...boo 0
...boo 1
...boo 2
...boo 3
...boo 4

问题

此实现是否通过了 asyncio 中并发循环任务的正确"示例?

Questions

Does this implementation pass for a 'proper' example of concurrent looping Tasks in asyncio?

唯一的方法是让 Task 提供阻塞点(await 表达式)以便事件循环处理多个任务，这是否正确?

Is it correct that the only way this works is for a Task to provide a blocking point (await expression) in order for the event loop to juggle multiple tasks?

推荐答案

是的，任何在您的事件循环中运行的协程都会阻止其他协程和任务运行，除非它

Yes, any coroutine that's running inside your event loop will block other coroutines and tasks from running, unless it

1. 使用 yield from 或 await(如果使用 Python 3.5+)调用另一个协程.
2. 退货.

1. Calls another coroutine using yield from or await (if using Python 3.5+).
2. Returns.

这是因为asyncio 是单线程的；事件循环运行的唯一方法是没有其他协程主动执行.使用 yield from/await 暂时挂起协程，让事件循环有机会工作.

This is because asyncio is single-threaded; the only way for the event loop to run is for no other coroutine to be actively executing. Using yield from/await suspends the coroutine temporarily, giving the event loop a chance to work.

您的示例代码很好，但在许多情况下，您可能不希望从不执行在事件循环内运行的异步 I/O 的长时间运行的代码开始.在这些情况下，使用 asyncio.loop.run_in_executor 在后台线程或进程中运行代码.如果您的任务受 CPU 限制，ProcessPoolExecutor 将是更好的选择，如果您需要执行一些不是 asyncio<的 I/O，将使用 ThreadPoolExecutor/code>-友好.

Your example code is fine, but in many cases, you probably wouldn't want long-running code that isn't doing asynchronous I/O running inside the event loop to begin with. In those cases, it often makes more sense to use asyncio.loop.run_in_executor to run the code in a background thread or process. ProcessPoolExecutor would be the better choice if your task is CPU-bound, ThreadPoolExecutor would be used if you need to do some I/O that isn't asyncio-friendly.

例如，您的两个循环完全受 CPU 限制并且不共享任何状态，因此最好的性能来自使用 ProcessPoolExecutor 跨 CPU 并行运行每个循环:

Your two loops, for example, are completely CPU-bound and don't share any state, so the best performance would come from using ProcessPoolExecutor to run each loop in parallel across CPUs:

import asyncio
from concurrent.futures import ProcessPoolExecutor

print('running async test')

def say_boo():
    i = 0
    while True:
        print('...boo {0}'.format(i))
        i += 1


def say_baa():
    i = 0
    while True:
        print('...baa {0}'.format(i))
        i += 1

if __name__ == "__main__":
    executor = ProcessPoolExecutor(2)
    loop = asyncio.get_event_loop()
    boo = loop.run_in_executor(executor, say_boo)
    baa = loop.run_in_executor(executor, say_baa)

    loop.run_forever()

这篇关于如何使用 python 的 asyncio 模块正确创建和运行并发任务?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！