在Python中使用多重处理和请求进行并行发布请求 [英] parallel post requests using multiprocessing and requests in Python
问题描述
我有如下的小代码段:
import requests
import multiprocessing
header = {
'X-Location': 'UNKNOWN',
'X-AppVersion': '2.20.0',
'X-UniqueId': '2397123',
'X-User-Locale': 'en',
'X-Platform': 'Android',
'X-AppId': 'com.my_app',
'Accept-Language': 'en-ID',
'X-PushTokenType': 'GCM',
'X-DeviceToken': 'some_device_token'
}
BASE_URI = 'https://my_server.com/v2/customers/login'
def internet_resource_getter(post_data):
stuff_got = []
response = requests.post(BASE_URI, headers=header, json=post_data)
stuff_got.append(response.json())
return stuff_got
tokens = [{"my_token":'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26Bpw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0esW3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBOxsaz'},{"my_token":'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAxZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZAwQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmTv3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD'}]
pool = multiprocessing.Pool(processes=3)
pool_outputs = pool.map(internet_resource_getter, tokens)
pool.close()
pool.join()
我要做的就是将并行的POST请求激发到终点,而每个POST都将具有不同的令牌作为其正文.
All I am trying to do is fire parallel POST requests to the end point, while each POST would have a different token as it's post body.
- 通过以上内容我将能够实现我想要的目标吗?我得到了输出,但是不确定我的请求是否并行发送.
- 我知道grequests.我想实现真正的并行请求(如利用系统上的多个处理器),因此我选择了多处理而不是grequest(据我所知,它使用的是gevents,它又不是并行的,而是多线程的).我的理解在这里正确吗?
推荐答案
如果您对并行执行多个POST请求感兴趣,建议您使用 aiohttp ,两者都实现了异步任务的概念,该任务可以并行运行.
If you interested in a parallel execution of multiple POST requests, I suggest you to use asyncio or aiohttp, which both implements the idea of asynchronous tasks, which run in parallel.
例如,您可以使用asyncio
做这样的事情:
For example, you can do something like this with asyncio
:
import requests
import asyncio
header = {
'X-Location': 'UNKNOWN',
'X-AppVersion': '2.20.0',
'X-UniqueId': '2397123',
'X-User-Locale': 'en',
'X-Platform': 'Android',
'X-AppId': 'com.my_app',
'Accept-Language': 'en-ID',
'X-PushTokenType': 'GCM',
'X-DeviceToken': 'some_device_token'
}
BASE_URI = 'https://my_server.com/v2/customers/login'
def internet_resource_getter(post_data):
stuff_got = []
response = requests.post(BASE_URI, headers=header, json=post_data)
stuff_got.append(response.json())
print(stuff_got)
return stuff_got
tokens = [
{
"my_token": 'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26B'
'pw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP'
'2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0es'
'W3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBO'
'xsaz'
},
{
"my_token": 'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAx'
'ZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZA'
'wQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmT'
'v3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD'
}
]
loop = asyncio.get_event_loop()
for token in tokens:
loop.run_in_executor(None, internet_resource_getter, token)
请注意:它们仅存在于python 3.x
中.但是,我认为它看起来更好,更简洁,并且可以确保它们并行运行.
Note this: They only exist in python 3.x
. But, it's look much better and concise in my opinion, and it's insure that they run in parallel.
这篇关于在Python中使用多重处理和请求进行并行发布请求的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!