如何使用python请求执行限时响应下载? [英] How to perform time limited response download with python requests?
问题描述
在用python下载大文件的时候,我不仅要对连接过程设置时间限制,还要对下载设置时间限制.
When downloading a large file with python, I want to put a time limit not only for the connection process, but also for the download.
我正在尝试使用以下 python 代码:
I am trying with the following python code:
import requests
r = requests.get('http://ipv4.download.thinkbroadband.com/1GB.zip', timeout = 0.5, prefetch = False)
print r.headers['content-length']
print len(r.raw.read())
这不起作用(下载没有时间限制),正如文档中正确指出的:https://requests.readthedocs.org/en/latest/user/quickstart/#timeouts
This does not work (the download is not time limited), as correctly noted in the docs: https://requests.readthedocs.org/en/latest/user/quickstart/#timeouts
如果可能的话,那就太好了:
This would be great if it was possible:
r.raw.read(timeout = 10)
问题是,如何设置下载时间限制?
The question is, how to put a time limit to the download?
推荐答案
答案是:不要使用请求,因为它会阻塞.使用非阻塞网络 I/O,例如 eventlet:
And the answer is: do not use requests, as it is blocking. Use non-blocking network I/O, for example eventlet:
import eventlet
from eventlet.green import urllib2
from eventlet.timeout import Timeout
url5 = 'http://ipv4.download.thinkbroadband.com/5MB.zip'
url10 = 'http://ipv4.download.thinkbroadband.com/10MB.zip'
urls = [url5, url5, url10, url10, url10, url5, url5]
def fetch(url):
response = bytearray()
with Timeout(60, False):
response = urllib2.urlopen(url).read()
return url, len(response)
pool = eventlet.GreenPool()
for url, length in pool.imap(fetch, urls):
if (not length):
print "%s: timeout!" % (url)
else:
print "%s: %s" % (url, length)
产生预期结果:
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
这篇关于如何使用python请求执行限时响应下载?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!