并行下载多个文件？ （Linux的/ Python的？） [英] Download many files in parallel? (Linux/Python?)

问题描述

我有一个很大的远程文件位置列表和本地路径，我希望他们结束。每个文件都很小，但是其中很多都是。我在Python中生成这个列表。

在打包和处理之前，我想尽快（并行）下载所有这些文件。什么是我最好的图书馆或linux命令行实用程序？我试图使用multiprocessing.pool实现这一点，但是这并不适用于FTP库。

我研究了pycurl，这似乎是我想要的，但是我无法让它在Windows 7 x64上运行。

解决方案

知道这是一个老帖子，但是有一个完美的linux实用程序这个。如果要从远程主机传输文件， lftp 是非常好的！我主要使用它来快速将东西推送到我的ftp服务器，但它也很好用于使用 mirror 命令拉出东西。它还有一个选项，可以按照您想要的方式并行复制用户定义的文件数。如果要将某些文件从远程路径复制到本地路径，您的命令行将如下所示：

  lftp 
 open ftp：// user：[e-mail] 
 cd some / remote / path 
 lcd some / local / path 
 mirror --reverse --parallel = 2 
  

尽管如此，非常小心这个命令，就像其他镜像命令，如果你拧了，你会删除文件。

有关 lftp 的更多选项或文档，我访问过此网站 http://lftp.yar.ru/lftp-man.html

I have a big list of remote file locations and local paths where I would like them to end up. Each file is small, but there are very many of them. I am generating this list within Python.

I would like to download all of these files as quickly as possible (in parallel) prior to unpacking and processing them. What is the best library or linux command-line utility for me to use? I attempted to implement this using multiprocessing.pool, but that did not work with the FTP library.

I looked into pycurl, and that seemed to be what I wanted, but I could not get it to run on Windows 7 x64.

解决方案

Know it's an old post but there is a perfect linux utility for this. If you are transferring files from a remote host, lftp is great! I mainly use it to quickly push stuff to my ftp server but it works great for pulling stuff off as well using the mirror command. It also has an option to copy a user defined number of files in parallel like you wanted. If you wanted to copy some files from a remote path to a local path your command line would look something like this;

lftp
open ftp://user:[e-mail]
cd some/remote/path
lcd some/local/path
mirror --reverse --parallel=2

Be very careful with this command though, just like other mirror commands if you screw it up, you WILL DELETE FILES.

For more options or documentation for lftp I've visited this site http://lftp.yar.ru/lftp-man.html

这篇关于并行下载多个文件？ （Linux的/ Python的？）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！