Python中的异步列表推导 [英] Async List Comprehensions in Python

问题描述

是否可以在Python中异步进行列表理解?我正在尝试对Web API进行多次调用，其中大部分等待时间都在等待数据返回，而这是我想异步进行的.然后，在提取数据之后，我想返回数据并将其存储在变量中.我将如何去做?

Is it possible to do a list comprehension in Python asynchronously? I am trying to make many calls to a web api, where the bulk of the wait time is waiting for data to come back, which I would like to do asynchronously. Then after the data is fetched, I want to return the data and store it in a variable. How would I go about doing this?

以下是非异步代码:

api_data = [{data_name: get_data(data_name)} for data_name in data_name_list]

以下是我尝试的异步解决方案:

Below is my attempted asynchronous solution:

async def get_api_data(data_name_list):
    api_data = [{data_name: get_data(data_name)} for data_name in data_name_list]
    await asyncio.wait(api_data)
    await conn.close()
    return api_data

data = get_api_data(data_name_list)

问题是 api_data 作为一个协程对象返回.自从我关闭连接后，它不应该不作为协程对象返回吗?有没有更好的方法可以将我的非异步代码转换为异步解决方案?

The problem is that the api_data gets returned as a coroutine object. Shouldn't it not return as a coroutine object since I close the connection? Is there a better way to turn my non asynchronous code into an asynchronous solution?

推荐答案

是的，确实有可能触发多个请求并异步等待它们.由于Python传统上是一种同步语言，因此您必须非常小心与异步Python一起使用的库.任何阻塞主线程的库(例如 requests )都会破坏您的整个异步性. aiohttp 是在Python中异步进行Web API调用的常见选择.您想要的是在Python列表内创建一堆 future 对象，然后等待它.未来是一个对象，它代表最终将解决的事物的价值.

Yes, it is indeed possible to fire off multiple requests and asynchronously wait for them. Because Python is traditionally a synchronous language, you have to be very careful about what libraries you use with asynchronous Python. Any library that blocks the main thread (such as requests) will break your entire asynchronicity. aiohttp is a common choice for asynchronously making web API calls in Python. What you want is to create a bunch of future objects inside a Python list and await it. A future is an object that represents a value that will eventually resolve to something.

由于实际进行API调用的函数是同步且阻塞的，并且您无法对其进行控制，因此必须在单独的线程中运行该函数.

Since the function that actually makes the API call is synchronous and blocking and you don't have control over it, you will have to run that function in a separate thread.

import asyncio

async def main():
    loop = asyncio.get_event_loop()
    futures = [asyncio.ensure_future(loop.run_in_executor(None, get_data, data)) for data in data_name_list]
    await asyncio.gather(*futures) # wait for all the future objects to resolve
    # Do something with futures
    # ...


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

请注意，这是Python 3.6代码，可能与3.7+异步代码大不相同

Please note that this is Python 3.6 code and might vary significantly from 3.7+ async code

这篇关于Python中的异步列表推导的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！