wget并行下载文件并重命名 [英] wget download files in parallel and rename

问题描述

我有一个包含两列的文本文件:第一列是要另存为的名称，第二列是资源的网址.

I have a text file with two columns: the first column is the name to be saved as, and the second column is the url address to the resource.

10000899567110806314.jpg 'http://lifestyle.inquirer.net/files/2018/07/t0724cheekee-marcopolo_1-e1532358505274-620x298.jpg'
10001149035013559957.jpg 'https://www.politico.eu/wp-content/uploads/2018/07/GettyImages-1004567890.jpg'
10001268622353586394.jpg 'http://www.channelnewsasia.com/image/10549912/16x9/991/529/a7afd249388308118058689b0060a978/Zv/tour-de-france-5.jpg'
10001360495981714191.jpg 'https://media.breitbart.com/media/2018/07/Dan-Coats.jpg'

该文件包含数千行，因此我希望有一种快速的方法来下载和重命名这些图像.

The file contains thousands of lines, so I wanted a quick way to download and rename these images.

我阅读了SO上的多个帖子，并提出了以下解决方案:

I read multiple posts on SO and came up with this solution:

cat list.txt  | xargs -n 1 -P 4 -d '\n' wget -O 

使用 xargs 并行下载.我想使用带有 -O 选项的 wget 重命名下载的文件.当我运行一个 wget 命令时，这很好.示例:

Which uses xargs to download in parallel. I want to use wget with -O option to rename the downloaded file. When I run a single wget command, this works well. Example:

wget -O 10000899567110806314.jpg 'http://lifestyle.inquirer.net/files/2018/07/t0724cheekee-marcopolo_1-e1532358505274-620x298.jpg'

但是在运行带有xargs的命令以并行下载时，出现此错误:

but when running the command with xargs to download in parallel, I get this error:

Try `wget --help' for more options.
wget: missing URL
Usage: wget [OPTION]... [URL]...

如果我生成的文件仅包含(单个col)换行符分隔的url，并运行以下命令，则效果很好.

If I generate a file with just (single col) newline delimited urls and run the following command, it works great.

cat list.txt  | xargs -n 1 -P 4 -d '\n' wget

但是，我不想先下载文件然后再执行重命名操作.

But, I don't want to download the files first and then do the rename operation.

推荐答案

您收到的错误是因为您只传递了一个参数 -n 1 使其起作用，您需要传递2参数，请尝试以下方法:

The error you are getting is because you are only passing one argument -n 1 to make it work you need to pass the 2 arguments, try this:

cat list.txt | xargs -n 2 -P 4 wget -O

要将全行用作@Pesa作为参数建议您可以使用选项 -L 1 ，例如:

To use the full line as an argument as @PesaThe suggested you could use option -L 1, for example:

xargs < list.txt -P 4 -L 1 wget -O

从男人那里来:

 -L number
     Call utility for every number non-empty lines read. 
     A line ending with a space continues to the next non-empty line. 
     If EOF is reached and fewer lines have been read than number then utility 
     will be called with the available lines.  The -L and -n options are
     mutually-exclusive; the last one given will be used.

这篇关于wget并行下载文件并重命名的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！