使用Curl命令行实用程序进行并行下载 [英] Parallel download using Curl command line utility

问题描述

我想从网站下载一些网页，我成功使用 curl 但我想知道如果某种方式 curl 一次下载多个页面，就像大多数下载管理器一样，它会加快一些事情。是否可以在 curl 命令行实用程序中执行？

I want to download some pages from a website and I did it successfully using curl but I was wondering if somehow curl downloads multiple pages at a time just like most of the download managers do, it will speed up things a little bit. Is it possible to do it in curl command line utility?

我使用的当前命令是

curl 'http://www...../?page=[1-10]' 2>&1 > 1.html

这里我从1到10下载页面，并将它们存储在一个名为 1.html 。

Here I am downloading pages from 1 to 10 and storing them in a file named 1.html.

此外， curl 将每个URL的输出写入单独的文件 URL.html ，其中 URL 。

Also, is it possible for curl to write output of each URL to separate file say URL.html, where URL is the actual URL of the page under process.

推荐答案

好吧， curl 只是一个简单的UNIX进程。您可以让这些 curl 进程并行运行，并将其输出发送到不同的文件。

Well, curl is just a simple UNIX process. You can have as many of these curl processes running in parallel and sending their outputs to different files.

curl 可以使用URL的文件名部分来生成本地文件。只需使用 -O 选项（ man curl 了解详情）。

curl can use the filename part of the URL to generate the local file. Just use the -O option (man curl for details).

您可以使用以下类似的内容

You could use something like the following

urls="http://example.com/?page1.html http://example.com?page2.html" # add more URLs here

for url in $urls; do
   # run the curl job in the background so we can start another job
   # and disable the progress bar (-s)
   echo "fetching $url"
   curl $url -O -s &
done
wait #wait for all background jobs to terminate

这篇关于使用Curl命令行实用程序进行并行下载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！