与从bash中的文件读取的数组并行的while循环 [英] Parallelizing a while loop with arrays read from a file in bash

问题描述

我在Bash中有一个while循环，它是这样处理的:

while IFS=$'\t' read -r -a line;
do
    myprogram ${line[0]} ${line[1]} ${line[0]}_vs_${line[1]}.result;
done < fileinput

它从具有这种结构的文件中读取，以供参考:

foo   bar
baz   foobar

以此类推(制表符分隔).

我想使用GNU parallel来并行化此循环(因为条目很多，并且处理可能很慢)，但是关于如何将每一行分配给数组的示例不清楚，就像我在这里所做的那样. /p>

什么是可能的解决方案(也可以替代GNU并行工作)?

解决方案

来自 table_file.tsv的内容:

foo<TAB>bar
baz <TAB> quux

要运行:

cmd -o bar -i foo
cmd -o quux -i baz

您可以运行:

parallel -a table_file.tsv --colsep '\t' cmd -o {2} -i {1}

""

因此，您的情况将是:

cat fileinput | parallel --colsep '\t' myprogram {1} {2} {1}_vs_{2}.result

I have a while loop in Bash handled like this:

while IFS=$'\t' read -r -a line;
do
    myprogram ${line[0]} ${line[1]} ${line[0]}_vs_${line[1]}.result;
done < fileinput

It reads from a file with this structure, for reference:

foo   bar
baz   foobar

and so on (tab-delimited).

I would like to parallelize this loop (since the entries are a lot and processing can be slow) using GNU parallel, however the examples are not clear on how I would assign each line to the array, like I do here.

What would be a possible solution (alternatives to GNU parallel work as well)?

解决方案

From https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Use-a-table-as-input:

"""
Content of table_file.tsv:

foo<TAB>bar
baz <TAB> quux

To run:

cmd -o bar -i foo
cmd -o quux -i baz

you can run:

parallel -a table_file.tsv --colsep '\t' cmd -o {2} -i {1}

"""

So in your case it will be:

cat fileinput | parallel --colsep '\t' myprogram {1} {2} {1}_vs_{2}.result

这篇关于与从bash中的文件读取的数组并行的while循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！