通过Python发送多个HTTP请求的理想方法？ [英] Ideal method for sending multiple HTTP requests over Python?

问题描述

可能重复：

与urllib2或其他http库的多个（异步）连接？

我在Linux web服务器上工作，运行Python代码，通过HTTP从第三方API获取实时数据。数据被放入MySQL数据库。
我需要对很多URL进行很多查询，我需要做快速（更快=更好）。目前我使用urllib3作为我的HTTP库。
什么是最好的办法呢？我应该产生多个线程（如果是，多少？），并让每个查询不同的URL？
我很想听听你的想法 - 谢谢！

解决方案

如果 >

$

$

kennethreitz / requests>请求 + gevent = grequests

GRequests允许您使用具有Gevent的Requests轻松地进行异步HTTP请求。

  import grequests 
 
 urls = [
'http://www.heroku.com'，
'http：// tablib .org'，
'http://httpbin.org'，
'http://python-requests.org'，
'http://kennethreitz.com'
] 
 
 rs =（grequests.get（u）for u in urls）
 grequests.map（rs）
  

Possible Duplicate:
Multiple (asynchronous) connections with urllib2 or other http library?

I am working on a Linux web server that runs Python code to grab realtime data over HTTP from a 3rd party API. The data is put into a MySQL database. I need to make a lot of queries to a lot of URL's, and I need to do it fast (faster = better). Currently I'm using urllib3 as my HTTP library. What is the best way to go about this? Should I spawn multiple threads (if so, how many?) and have each query for a different URL? I would love to hear your thoughts about this - thanks!

解决方案

If a lot is really a lot than you probably want use asynchronous io not threads.

requests + gevent = grequests

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

import grequests

urls = [
    'http://www.heroku.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
grequests.map(rs)

这篇关于通过Python发送多个HTTP请求的理想方法？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！