将具有阻塞请求的库重写为异步请求 [英] Rewrite a library with blocking requests into async ones

问题描述

有一个库在其核心使用阻塞requests，我想将其重写为异步版本，所以请您建议最好/最简单的策略是什么.

There is a library that uses blocking requests in its core and I would like to rewrite it into asynchronous version, so could you please advise what would be the best/easiest strategy to do so.

整个库，经过几个嵌套函数，调用一个函数:

The whole library, after several nested functions, calls one function:

def _send_http_request(self, url, payload, method='post', **kwargs):
# type: (Text, Optional[Text], Text, dict) -> Response

response = request(method=method, url=url, data=payload, **kwargs)

return response

仅仅将 async 放在它前面是行不通的，因为它深深地嵌套在阻塞函数中.重写所有内容会太麻烦.我查看了 aiohttp、trio、asks 并且有点迷路了，哪个更好.我知道 celery 或 dask，但我需要异步.

Just putting async in front of it wont work since it is deeply nested in blocking functions. And rewriting everything would be a way too much hassle. I had a look into aiohttp, trio, asks and kinda got lost, which one is better. I know about celery or dask, but I need async.

推荐答案

您有几个选择:

1. 将 _send_http_request 重写为异步(例如，使用 aiohttp) 并进一步将所有使用 _send_http_request 的函数重写为异步函数.是的，还有很多工作要做，但这就是 asyncio 的基本设计方式.

1. Rewrite _send_http_request to be async (using, for example, aiohttp) and further rewrite all functions that use _send_http_request to be async either. Yes, it's much work to do, but this is how asyncio fundamentally designed.

仅包装您需要使用 run_in_executor 如此处.如果您不打算发出数百万个请求，您将看不到上述选项的性能差异，因为主要瓶颈仍然是 I/O.否则，与纯 asyncio 解决方案相比，线程开销会很明显.

Wrap only top-level blocking functions (functions with I/O) you need to run asynchronously with run_in_executor as explained here. If you aren't going to make millions of requests you won't see much performance difference with option above since main bottleneck is still I/O. Otherwise threads overhead will be noticeable compared to pure asyncio solution.

尝试其他解决方案而不是 asyncio.例如，gevent 及其 猴子补丁.这种方法各有利弊.

Try other solution instead of asyncio. For example, gevent and its monkey-patching. This approach has own pros and cons.

这篇关于将具有阻塞请求的库重写为异步请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！