通过 Python 发送多个 HTTP 请求的理想方法? [英] Ideal method for sending multiple HTTP requests over Python?

问题描述

可能的重复:
与 urllib2 或其他 http 库的多个(异步)连接?

我正在开发一个 Linux Web 服务器，该服务器运行 Python 代码以通过 HTTP 从 3rd 方 API 获取实时数据.数据被放入 MySQL 数据库中.我需要对很多 URL 进行大量查询，而且我需要快速(更快 = 更好).目前我使用 urllib3 作为我的 HTTP 库.解决这个问题的最佳方法是什么?我应该生成多个线程(如果是，有多少?)并让每个查询都使用不同的 URL?我很想听听您对此的看法 - 谢谢！

I am working on a Linux web server that runs Python code to grab realtime data over HTTP from a 3rd party API. The data is put into a MySQL database. I need to make a lot of queries to a lot of URL's, and I need to do it fast (faster = better). Currently I'm using urllib3 as my HTTP library. What is the best way to go about this? Should I spawn multiple threads (if so, how many?) and have each query for a different URL? I would love to hear your thoughts about this - thanks!

推荐答案

如果很多确实很多，您可能希望使用异步 io 而不是线程.

If a lot is really a lot than you probably want use asynchronous io not threads.

请求 + gevent = grequests

GRequests 允许您将 Requests 与 Gevent 结合使用，以轻松生成异步 HTTP 请求.

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

import grequests

urls = [
    'http://www.heroku.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
grequests.map(rs)

这篇关于通过 Python 发送多个 HTTP 请求的理想方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！