Python:如何同时发送多个http请求?(如叉子) [英] Python: How can i send multiple http requests at the same time? (like fork)
问题描述
假设我有一种方法可以将 http请求发送到服务器.如何将其中两个(或多个)请求同时发送到服务器?例如，也许通过分叉过程?我该怎么做?(也是我在用django)
Let's say that i have a way to send http request to a server. How it's possible to send two of these requests (or more) to the server at the same time? For example maybe by fork a process? How can i do it? (also i'm using django)
#This example is not tested...
import requests
def tester(request):
server_url = 'http://localhost:9000/receive'
payload = {
'd_test2': '1234',
'd_test2': 'demo',
}
json_payload = simplejson.dumps(payload)
content_length = len(json_payload)
headers = {'Content-Type': 'application/json', 'Content-Length': content_length}
response = requests.post(server_url, data=json_payload, headers=headers, allow_redirects=True)
if response.status_code == requests.codes.ok:
print 'Headers: {}\nResponse: {}'.format(response.headers, response.text)
谢谢！
推荐答案
我认为您想在这里使用线程，而不是分叉新进程.尽管在某些情况下线程很糟糕，但在这里并非如此.另外，我认为您想使用 concurrent.futures 而不是直接使用线程(或进程).
I think you want to use threads here rather than forking off new processes. While threads are bad in some cases, that isn't true here. Also, I think you want to use concurrent.futures instead of using threads (or processes) directly.
例如，假设您有10个URL，并且您当前正在连续执行这些操作，如下所示:
For example, let's say you have 10 URLs, and you're currently doing them one in a row, like this:
results = map(tester, urls)
但是现在，您想一次发送给他们2个.只需将其更改为此:
But now, you want to send them 2 at a time. Just change it to this:
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
results = pool.map(tester, urls)
如果您想一次尝试4次而不是2次，只需更改 max_workers .实际上，您可能应该尝试使用不同的值，以查看最适合您的程序的
If you want to try 4 at a time instead of 2, just change the max_workers. In fact, you should probably experiment with different values to see what works best for your program.
如果您想做一些更简单的事情，请参阅文档-主要的 ThreadPoolExecutor示例几乎正是您所需要的.
If you want to do something a little fancier, see the documentation—the main ThreadPoolExecutor Example is almost exactly what you're looking for.
不幸的是，在2.7中，该模块不随标准库一起提供，因此您必须安装PyPI的反向端口.
Unfortunately, in 2.7, this module doesn't come with the standard library, so you will have to install the backport from PyPI.
如果您已安装 pip ，则应该就这么简单:
If you have pip installed, this should be as simple as:
pip install futures
在Unix上
…或 sudo pip安装期货.
如果您没有 pip ，请先获取它(按照上面的链接).
And if you don't have pip, go get it first (follow the link above).
您有时想要使用进程而不是线程的主要原因是您拥有大量CPU限制的计算，并且您希望利用多个CPU内核.在Python中，线程无法有效地耗尽您的所有内核.因此，如果任务管理器"/活动监视器"/无论什么显示您的程序在一个内核上使用了100％的CPU，而其他内核全部使用了0％，则进程就是答案.使用 futures ，只需将 ThreadPoolExecutor 更改为 ProcessPoolExecutor .
The main reason you sometimes want to use processes instead of threads is that you've got heavy CPU-bound computation, and you want to take advantage of multiple CPU cores. In Python, threading can't effectively use up all your cores. So, if the Task Manager/Activity Monitor/whatever shows that your program is using up 100% CPU on one core, while the others are all at 0%, processes are the answer. With futures, all you have to do is change ThreadPoolExecutor to ProcessPoolExecutor.
与此同时，有时您不仅需要给我一个神奇的工作池来执行任务".有时，您想运行几个非常长的工作，而不是一堆小工作，或者自己对工作进行负载平衡，或者在工作之间传递数据，等等.为此，您要使用 multiprocessing 或<一个href ="http://docs.python.org/2/library/threading.html" rel ="noreferrer"> threading 而不是 futures .
Meanwhile, sometimes you need more than just "give me a magic pool of workers to run my tasks". Sometimes you want to run a handful of very long jobs instead of a bunch of little ones, or load-balance the jobs yourself, or pass data between jobs, or whatever. For that, you want to use multiprocessing or threading instead of futures.
非常罕见，甚至 太高级了，它直接告诉Python创建一个新的子进程或线程.为此，请一直进行到 os.fork (仅在Unix上)或 thread .
Very rarely, even that is too high-level, and directly tell Python to create a new child process or thread. For that, you go all the way down to os.fork (on Unix only) or thread.
这篇关于Python:如何同时发送多个http请求?(如叉子)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
