提高请求的下载速度 [英] Increase download speed of requests

问题描述

我正在编写一个脚本，用于从 Dropbox 下载一些视频.通常我的下载速度约为 150 kb/秒，这是在 Firefox 或 IDM 上使用普通下载器时的速度.但是在使用这个 Python 脚本时，事情变得太慢了:不到 10 kb/秒.

I'm writing a script that download some videos from dropbox. Usually my downloading speed is around 150 kb/sec, this when using normal downloader on firefox or IDM. But while using this Python script, things get too slow: less than 10 kb / sec.

这是我正在使用的代码:

Here's the code I'm using :

def download(url,i):
    local_filename = "video_" + str(i) + ".mp4"
    # NOTE the stream=True parameter
    r = requests.get(url, stream=True)
    with open(local_filename, 'wb') as f:
        n = 0
        for chunk in r.iter_content(chunk_size=1000000): 
            if chunk: # filter out keep-alive new chunks
                n = n +1
                f.write(chunk)
                print "downloading " + str(n)
                f.flush() 
    return local_filename

无论如何我可以加快下载速度吗?

Is there anyway I could speed up the download?

推荐答案

我认为一次加载 1000000mb 是一个很大的值.当所有数据都缓冲到内存中时，您无法对块做任何事情.如果文件大小小于 512mb，我会将大小减小到 512mb 或 1gb，那么块大小应该小于该大小，否则您不是真正加载块，而是整个文件......仍然如果您搜索网络你会看到它仍然是一个热门话题辩论.示例代码:

I believe that 1000000mb is to large of a value to load at one time. You can't do anything with chunk while all that data is buffering into memory. I would reduce the size to 512mb or 1gb if the file size is smaller than 512mb then the chunk size should be smaller than that, otherwise you're not really loading chunks, but the entire file...still if you search the net a bit you'll see that its a hot topic debate still. Example code:

def download(url,i):
    local_filename = "video_" + str(i) + ".mp4"
    with open(local_filename, 'wb') as f, closing(requests.get(url, stream=True)) as res:
        for n, chunk in enumerate(res.iter_content(chunk_size=512), start=1):
          f.write(chunk)
          print "downloading " + str(n)
    return local_filename

advance docs 建议在流式传输时将其包装在结束语境.它应该稳定请求.if 块不需要过滤掉keep alive 也不需要flush，直接写入文件流即可.

The advance docs suggest that when streaming to wrap it in the closing context. It should stabilize requests. The if chunk is not needed to filter out keep alive nor do you have to flush, just write to the file stream directly.

这篇关于提高请求的下载速度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！