aiohttp设置每秒请求数 [英] aiohttp set number of requests per second

问题描述

我正在Flask中编写一个API，其中包含1000多个获取数据的请求，我想限制每秒的请求数.我尝试过:

I'm writing an API in Flask with 1000+ requests to get data and I'd like to limit the number of requests per second. I tried with:

conn = aiohttp.TCPConnector(limit_per_host=20)

and 

conn = aiohttp.TCPConnector(limit=20)

但是似乎不起作用

我的代码如下:

import logging
import asyncio
import aiohttp

logging.basicConfig(filename="logfilename.log", level=logging.INFO, format='%(asctime)s %(levelname)s:%(message)s')


async def fetch(session, url):
    async with session.get(url, headers=headers) as response:
        if response.status == 200:
            data = await response.json()
            json = data['args']
        return json

async def fetch_all(urls, loop):
    conn = aiohttp.TCPConnector(limit=20)
    async with aiohttp.ClientSession(connector=conn, loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results

async def main():
    loop = asyncio.new_event_loop()
    url_list = []
    args = ['a', 'b', 'c', +1000 others]
    urls = url_list
    for i in args:
        base_url = 'http://httpbin.org/anything?key=%s' % i
        url_list.append(base_url)

    htmls = loop.run_until_complete(fetch_all(urls, loop))
    for j in htmls:
        key = j['key']
        # save to database
        logging.info(' %s was added', key)

如果我运行代码，则在1秒内我发送了200多个请求.有什么办法可以限制请求?

If I run code, within 1s I send over than 200 requests. Is there any way to limit requests?

推荐答案

上面的代码按预期工作(除了有关 headers 未定义的小错误).

The code above works as expected (apart from a small error regarding headers being undefined).

在我的计算机上进行测试， httpbin URL大约在100毫秒内响应，这意味着并发性为20的URL将在1秒内处理大约200个请求(这也是您所看到的):

Tested on my machine the httpbin URL responds in around 100ms which means that with a concurrency of 20 it will serve around 200 requests in 1 second (which is what you're seeing as well):

100毫秒表示 10个请求在一秒钟内完成
每秒 10个请求和并发性 20 表示在一秒钟内 200个请求

100 ms per request means 10 requests are completed in a second
10 requests per second with a concurrency of 20 means 200 requests in one second

限制选项( aiohttp.TCPConnector )限制并发请求的数量，并且没有任何时间维度.

The limit option (aiohttp.TCPConnector) limits the number of concurrent requests and does not have any time dimension.

要查看实际限制，请尝试使用更多值，例如 10 ， 20 ， 50 :

To see the limit in action try with more values like 10, 20, 50:

# time to complete 1000 requests with different keys
aiohttp.TCPConnector(limit=10): 12.58 seconds 
aiohttp.TCPConnector(limit=20): 6.57 seconds
aiohttp.TCPConnector(limit=50): 3.1 seconds

如果您想使用每秒 requests 的限制来发送一批请求(例如20个)，然后使用 asyncio.sleep(1.0)暂停一秒钟，然后发送下一批，依此类推.

If you want to use a requests per second limit send batch of requests (20 for example) and use asyncio.sleep(1.0) to pause for a second, then send the next batch and so on.

这篇关于aiohttp设置每秒请求数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！