从bash中的txt文件中多次读取(并行处理) [英] Multiple read from a txt file in bash (parallel processing )

问题描述

这是一个用于HTTP状态代码的简单bash脚本

Here is a simple bash script for HTTP status code

while read url
    do
        urlstatus=$(curl -o /dev/null --silent --head --write-out  '%{http_code}' "${url}" --max-time 5 )
        echo "$url  $urlstatus" >> urlstatus.txt
    done < $1

我正在从文本文件中读取URL，但是它一次只能处理一个，占用太多时间，GNU并行和xargs一次也只能处理一行(已测试)

I am reading URL from text file but it processes only one at a time, taking too much time, GNU parallel and xargs also process one line at time (tested)

如何处理同时URL以进行处理以缩短时间?换句话说，URL文件的线程而不是bash命令(GNU parallel和xargs都这样做)

How to process simultaneous URL for processing to improve timing? In other words threading of URL file rather than bash commands (which GNU parallel and xargs do)

 Input file is txt file and lines are separated  as
    ABC.Com
    Bcd.Com
    Any.Google.Com

Something  like this

推荐答案

GNU并行和xargs一次也处理一行(经过测试)

GNU parallel and xargs also process one line at time (tested)

您能举个例子吗?如果使用 -j ，则一次应该可以运行多个进程.

Can you give an example of this? If you use -j then you should be able to run much more than one process at a time.

我会这样写:

doit() {
    url="$1"
    urlstatus=$(curl -o /dev/null --silent --head --write-out  '%{http_code}' "${url}" --max-time 5 )
    echo "$url  $urlstatus"
}
export -f doit
cat "$1" | parallel -j0 -k doit >> urlstatus.txt

基于输入:

Input file is txt file and lines are separated  as
ABC.Com
Bcd.Com
Any.Google.Com
Something  like this
www.google.com
pi.dk

我得到输出:

Input file is txt file and lines are separated  as  000
ABC.Com  301
Bcd.Com  301
Any.Google.Com  000
Something  like this  000
www.google.com  302
pi.dk  200

哪个看起来正确:

000 if domain does not exist
301/302 for redirection
200 for success

这篇关于从bash中的txt文件中多次读取(并行处理)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！