请求 response.iter_content() 获取不完整的文件( 1024MB 而不是 1.5GB )? [英] requests response.iter_content() gets incomplete file ( 1024MB instead of 1.5GB )?
问题描述
我一直在使用此代码片段从网站下载文件,到目前为止,小于 1GB 的文件都很好.但我注意到一个 1.5GB 的文件不完整
hi i have been using this code snippet to download files from a website, so far files smaller than 1GB are all good. but i noticed a 1.5GB file is incomplete
# s is requests session object
r = s.get(fileUrl, headers=headers, stream=True)
start_time = time.time()
with open(local_filename, 'wb') as f:
count = 1
block_size = 512
try:
total_size = int(r.headers.get('content-length'))
print 'file total size :',total_size
except TypeError:
print 'using dummy length !!!'
total_size = 10000000
for chunk in r.iter_content(chunk_size=block_size):
if chunk: # filter out keep-alive new chunks
duration = time.time() - start_time
progress_size = int(count * block_size)
if duration == 0:
duration = 0.1
speed = int(progress_size / (1024 * duration))
percent = int(count * block_size * 100 / total_size)
sys.stdout.write("\r...%d%%, %d MB, %d KB/s, %d seconds passed" %
(percent, progress_size / (1024 * 1024), speed, duration))
f.write(chunk)
f.flush()
count += 1
使用最新请求 2.2.1 python 2.6.6,centos 6.4文件下载总是在 66.7% 1024MB 处停止,我错过了什么?输出:
using latest requests 2.2.1 python 2.6.6, centos 6.4 the file download always stops at 66.7% 1024MB, what am i missing ? the output:
file total size : 1581244542
...67%, 1024 MB, 5687 KB/s, 184 seconds passed
iter_content() 返回的生成器似乎认为所有块都已检索并且没有错误.顺便说一句,异常部分没有运行,因为服务器确实在响应头中返回了内容长度.
it seems the generator returned by iter_content() thinks all chunks are retrieved and there is no error. btw the exception part did not run, because the server did return the content-length in response header.
推荐答案
请仔细检查您是否可以通过 wget
和/或任何常规浏览器下载文件.可能是服务器限制.正如我所见您的代码可以下载大文件(大于 1.5Gb)
Please double check that you can download the file via wget
and/or any regular browser. It could be restriction on the server. As I see your code can download big files (bigger then 1.5Gb)
更新:请尝试反转逻辑 - 而不是
Update: please try to inverse the logic - instead of
if chunk: # filter out keep-alive new chunks
f.write(chunk)
f.flush()
试试
if not chunk:
break
f.write(chunk)
f.flush()
这篇关于请求 response.iter_content() 获取不完整的文件( 1024MB 而不是 1.5GB )?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!