While循环中的并行处理 [英] Parallel Processing in While Loop

问题描述

我有一个.sh从.txt读取行以用作另一个shell脚本的参数，每两行并行处理. 当前:我的当前代码将在文件中的所有行(一次2个)中读取所有行，然后在第二个.sh调用中读取ALL行，然后在最后一个.sh调用中读取ALL

I have a .sh that is reading in lines from a .txt to be used as parameters for another shell script.Every two lines process in parallel. Currently: My current code will read in ALL lines in the file (2 at a time) for the first .sh call, then ALL lines for second .sh call, then ALL for the last .sh call

问题:我需要在第一个.sh中先行，然后在第二个.sh中，然后最后一个.sh..THEN循环返回，并用接下来的两行处理HHEEELLLPPP ！！！ :)

Problem: I need to first two lines in first .sh, then second .sh, then last .sh..THEN loop back and process with next two lines HHEEELLLPPP!!! :)

现在:

cat XXXXX.txt | while read line; do
export step=${line//\"/}
export step=ExecuteModel_${step//,/_}
export pov=$line

$dir"/hpm_ws_client.sh" processCalcScriptOptions  "$appName" "$pov" "$layers" "$stages" "" "$stages" "$stages" FALSE > "$dir""/"$step"_ProcessID.log"
$dir_shell_model"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &


$dir"/hpm_ws_client.sh" processCalcScriptOptions "$appName" "$pov" "$layers" "" ""  "$stage_final" "" TRUE > "$dir""/""$step""_ProcessID.log"
$dir"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &

$dir"/hpm_ws_client.sh" processGenealogyExecutionPaths "$appName" "$pov" "$layers" "$genPath_01" > "$dir""/""$step""_ProcessID.log"
$dir"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &

if (( ++i % 2 == 0))
then
echo "Waiting..."
wait
fi
done

推荐答案

我看不到您真正想做的事情，但希望这两种语法之一能有所帮助-一次读取两行，或加载参数变成一个数组，然后重新使用它们.

I cannot see what you are really trying to do, but hope one of these two syntaxes will help - either reading two lines at a time, or loading the parameters into an array an re-using them.

因此，如果您的file.txt看起来像这样:

So, if your file.txt looks like this:

line 1
line 2
line 3
line 4
line 5
line 6

示例1-两次读取

#!/bin/bash
while read a && read b; do
   echo $a, $b
done < file.txt

输出

line 1, line 2
line 3, line 4
line 5, line 6

示例2-具有bash数组

#!/bin/bash
declare -a params
while IFS=$'\n' read -r z; do
    params+=("${z}")
done < file.txt

# Now print the elements out
for (( i=0;i<${#params[@]};i++ )) do
   echo ${params[$i]}
done

输出

line 1
line 2
line 3
line 4
line 5
line 6

示例3-使用GNU并行

或者，正如我在评论中建议的那样，使用GNU Parallel这样

Or, as I suggested in my comment, use GNU Parallel like this

parallel -k -L2 echo {1} {2} < file.txt

输出

line 1 line 2
line 3 line 4
line 5 line 6

其中-k表示按顺序保留输出" ，而-L2表示每次从file.txt中提取两行" .

这样做的好处是，如果您希望一次并行运行8个脚本，只需将-j 8设置为parallel即可完成工作.

This has the advantage that, if you want to run 8 scripts at a time in parallel, you just specify -j 8 to parallel and the job is done.

这篇关于While循环中的并行处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！