aiohttp：设置每秒的最大请求数 [英] aiohttp: set maximum number of requests per second

问题描述

如何使用aiohttp在客户端设置每秒最大请求数（限制请求数）？

How can I set maximum number of requests per second (limit them) in client side using aiohttp?

推荐答案

一种可能的解决方案： http://compiletoi.net/fast-scraping-in-python-with -asyncio.html

I found one possible solution here: http://compiletoi.net/fast-scraping-in-python-with-asyncio.html

同时执行3个请求很酷，但同时执行5000个则不是很好。如果您尝试同时执行太多请求，则连接可能会开始关闭，甚至可能被网站禁止。

Doing 3 requests at the same time is cool, doing 5000, however, is not so nice. If you try to do too many requests at the same time, connections might start to get closed, or you might even get banned from the website.

为避免这种情况，您可以可以使用信号量。它是一个同步工具，可用于限制在某些时候执行某些操作的协程数量。我们只是在创建循环之前创建信号灯，将要允许的并发请求数作为参数传递：

To avoid this, you can use a semaphore. It is a synchronization tool that can be used to limit the number of coroutines that do something at some point. We'll just create the semaphore before creating the loop, passing as an argument the number of simultaneous requests we want to allow:

sem = asyncio.Semaphore(5)

然后，我们只需替换：

Then, we just replace:

page = yield from get(url, compress=True)

受同一事物影响，但受信号量保护：

by the same thing, but protected by a semaphore:

with (yield from sem):
    page = yield from get(url, compress=True)

这将确保最多可以请求5个同时完成。

This will ensure that at most 5 requests can be done at the same time.

这篇关于aiohttp：设置每秒的最大请求数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！