如何使用Python 2.7通过HTTP使用多线程下载文件(异步下载) [英] How to download a file over HTTP with multi-thread (asynchronous download) using Python 2.7
问题描述
我有一个文件要下载(从json中提取下载路径,例如:http://testsite/abc.zip
).
我需要执行帮助,所有5个线程都应将abc.zip
文件下载到输出目录,并且下载必须为异步或并发. /p>
当前使用以下代码,它确实可以下载文件5次,但可以一次下载一个(同步).
我想要的是同时下载.
def dldr(file=file_url, outputdir=out1):
local_fn = str(uuid.uuid4())
if not os.path.exists(outputdir):
os.makedirs(outputdir)
s = datetime.now()
urllib.urlretrieve(file, outputdir + os.sep + local_fn)
e = datetime.now()
time_diff = e - s
logger(out1, local_fn, time_diff)
for i in range(1, 6):
t = threading.Thread(target=dldr())
t.start()
我已阅读具有多个连接的请求,它很有帮助,但没有解决问题的要求.
我将线程模块用于下载线程:
也可以请求,但是您可以自己将其更改为urllib.
import threading
import requests
def download(link, filelocation):
r = requests.get(link, stream=True)
with open(filelocation, 'wb') as f:
for chunk in r.iter_content(1024):
if chunk:
f.write(chunk)
def createNewDownloadThread(link, filelocation):
download_thread = threading.Thread(target=download, args=(link,filelocation))
download_thread.start()
for i in range(0,5):
file = "C:\\test" + str(i) + ".png"
print file
createNewDownloadThread("http://stackoverflow.com/users/flair/2374517.png", file)
I have a file to download (download path extracted from json. eg: http://testsite/abc.zip
).
I need a help to perform, all the 5 threads should download the abc.zip
file to the output directory and the download has to be Asynchronous or concurrent.
Currently with the below code it does download the file 5 times but it downloads one by one (Synchronous).
What I want is, the download to be simultaneous.
def dldr(file=file_url, outputdir=out1):
local_fn = str(uuid.uuid4())
if not os.path.exists(outputdir):
os.makedirs(outputdir)
s = datetime.now()
urllib.urlretrieve(file, outputdir + os.sep + local_fn)
e = datetime.now()
time_diff = e - s
logger(out1, local_fn, time_diff)
for i in range(1, 6):
t = threading.Thread(target=dldr())
t.start()
I have read Requests with multiple connections post and it's helpful, but doesn't address the requirement of the question asked.
I use threading module for download threads:
Also requests, but you can change that to urllib by yourself.
import threading
import requests
def download(link, filelocation):
r = requests.get(link, stream=True)
with open(filelocation, 'wb') as f:
for chunk in r.iter_content(1024):
if chunk:
f.write(chunk)
def createNewDownloadThread(link, filelocation):
download_thread = threading.Thread(target=download, args=(link,filelocation))
download_thread.start()
for i in range(0,5):
file = "C:\\test" + str(i) + ".png"
print file
createNewDownloadThread("http://stackoverflow.com/users/flair/2374517.png", file)
这篇关于如何使用Python 2.7通过HTTP使用多线程下载文件(异步下载)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!