python3.6使用aiohttp和asyncio启动一百万个请求 [英] python3.6 start 1 million requests with aiohttp and asyncio
问题描述
我正在尝试通过aiohttp和asyncio连续发出10百万次请求，每次发出10k次，每次发出10k次。当我打印每个请求的开始时间时，我发现100万个请求不是在非常封闭的时间开始，而是在服务时间（分钟）内开始。以我的理解，这100万个请求将在没有等待的情况下发送（或仅以微秒为单位？）。希望有人可以帮助我提出如何更改代码的建议，我的代码如下。
I'm trying to make 1 million requests with aiohttp and asyncio continuously in 10 times which 10k at each time. When I print the start time of each request, I found that the 1 million requests are NOT start at a very closed time but in serval minutes. In my understanding, the 1 million requests will be sent without any wait(or just say in microseconds?) Hope someone can help me give a suggestion how to change the code, and my code is as below. Thanks in advance!
import asyncio
import requests
import json
import pymysql
from aiohttp import ClientSession
from datetime import datetime
import uvloop
# login config
URL_LOGIN = "https://test.com/user/login"
APP_ID = "sample_app_id"
APP_SECRET = "sample_secret"
async def login_user(phone, password, session, i):
start_time = datetime.now()
h = {
"Content-Type": "application/json"
}
data = {
"phone": phone,
"password": password,
"appid": APP_ID,
"appsecret": APP_SECRET
}
try:
async with session.post(url=URL_LOGIN, data=json.dumps(data), headers=h) as response:
r = await response.read()
end_time = datetime.now()
cost = (end_time-start_time).seconds
msg = "number %d request，start_time：%s, cost_time: %d, response: %s\n" % (i, start_time, cost, r.decode())
print("running %d" % i, datetime.now())
except Exception as e:
print("running %d" % i)
msg = "number %d request raise error" % i+str(e)+"\n"
with open("log", "a+") as f:
f.write(msg)
async def bound_login(sem, phone, password, session, i):
async with sem:
await login_user(phone, password, session, i)
async def run_login(num):
tasks = []
sem = asyncio.Semaphore(10000)
async with ClientSession() as session:
for i in range(num):
task = asyncio.ensure_future(bound_login(sem, str(18300000000+i), "123456", session, i))
tasks.append(task)
responses = asyncio.gather(*tasks)
await responses
start = datetime.now()
number = 100000
loop = uvloop.new_event_loop()
asyncio.set_event_loop(loop)
future = asyncio.ensure_future(run_login(number))
推荐答案
当我打印每个请求的开始时间时，我发现这100万个请求不是在非常封闭的时间开始，而是在服务分钟内。
When I print the start time of each request, I found that the 1 million requests are NOT start at a very closed time but in serval minutes.
您的代码确实发出了1百万个请求，但在任何给定时间，并行执行的请求不得超过1万个。这就像有1万个请求槽供您使用-前10,000个请求将立即开始，但第10,001个请求槽必须等待上一个请求完成，这样才能获得一个免费的槽。
Your code does issue a total of 1 million requests, but with the constraint that no more than 10 thousand of them runs in parallel at any given time. This is like having 10k request slots at your disposal - the first 10,000 requests will be started immediately, but the 10,001st will have to wait for a previous request to finish so it can get a free slot.
这就是为什么100万个请求无法立即或近乎瞬时启动的原因，其中大多数都必须等待下载完成，这需要时间。
This is why 1 million requests cannot start instantaneously or near-instantaneously, most of them have to wait for some download to finish, and that takes time.
在我的理解中，将发送100万个请求而无需等待
当前代码明确地使请求等待，以防止超过10k的请求并行运行。如果您确实要（尝试）发出一百万个并行请求，请删除该信号灯并使用限制设置为无。
The current code explicitly makes the requests wait in order to prevent more than 10k of them running in parallel. If you really want to (try to) make a million parallel requests, remove the semaphore and create the ClientSession using a connector with limit set to None.
但是，请注意，由于操作系统和硬件的限制，维持一百万个开放连接可能无法正常工作。 （您仍然应该能够近乎瞬时地启动连接，但是我希望它们中的大多数都将在不久后退出，但有例外。）
However, be aware that maintaining a million open connections will likely not work due to limits of the operating system and the hardware. (You should still be able to start the connections near-instantaneously, but I'd expect most of them to exit with an exception shortly afterwards.)
这篇关于python3.6使用aiohttp和asyncio启动一百万个请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
