在管道中使用 GNU 并行 [英] use GNU parallel in a pipe
问题描述
从文件读取时运行以下命令执行我想要的操作:
Running the following command does what I want when reading from a file:
parallel --gnu -j2 "echo {} && sleep 5" < myfile.txt
我想用管道做类似的事情.请注意,我使用以下页面为管道读取器和写入器提供灵感:http://www.linuxjournal.com/content/using-named-pipes-fifos-bash
I would like to do something similar with a pipe. Note that I used the following page for inspiration for the pipe reader and writer: http://www.linuxjournal.com/content/using-named-pipes-fifos-bash
这是我的管道读取器文件:
Here is my pipe reader file:
#!/bin/bash
pipe=/tmp/testpipe
trap "rm -f $pipe" EXIT
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
while true
do
parallel --gnu -j2 "echo {} && sleep 5" <$pipe
done
echo "Reader exiting"
这是我的作家:
#!/bin/bash
pipe=/tmp/testpipe
if [[ ! -p $pipe ]]; then
echo "Reader not running"
exit 1
fi
if [[ "$1" ]]; then
echo "$1" >$pipe
else
echo "Hello from $$" >$pipe
fi
然后我多次运行pipeWriter,例如
I then run pipeWriter several times, such as
$ ./pipeWriter one
$ ./pipeWriter two
$ ./pipeWriter three
$ ./pipeWriter four
$ ./pipeWriter five
$ ./pipeWriter six
$ ./pipeWriter seven
$ ./pipeWriter eight
$ ./pipeWriter nine
$ ./pipeWriter ten
$ ./pipeWriter ww
$ ./pipeWriter wwdfsdf
$ ./pipeWriter wwdfsdfsddfsd
$ ./pipeWriter testAgain
运行 pipeReader 的 shell 显示:
The shell running pipeReader shows:
$ ./pipeReader
one
four
eight
ww
testAgain
首先,存在数据缺失的问题.其次,parallel 在管道中似乎并没有并行运行.我希望它一次运行两个作业(或者更确切地说,最多两个作业.如果它只有一个工作,那很好,如果另一个它可以启动它).
First, there is a problem of missing data. Second, parallel does not seem to run in parallel when in the pipe. I would like it to run two jobs at a time (or rather, at most two jobs. If it only has one job that's fine and if another it can start it).
我哪里出错了?
推荐答案
我无法解释丢失的数据,并且当我运行它时我没有得到丢失的数据.但是,我可以解释为什么您看不到并行运行的作业:阅读器中的 while
循环可能仅将一个参数传递给 parallel
.
I cannot explain the missing data, and when I run it I do not get missing data. I can, however, explain why you do not see the jobs run in parallel: Your while
-loop in the reader may pass only one argument to parallel
.
使用 http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_queue_system_batch_manager
也许你也可以使用cat
:
while true
do
cat $pipe
done | parallel --gnu -j2 "echo {} && sleep 5"
如果您还没有这样做,请完成本教程;你的命令行会喜欢你的:http://www.gnu.org/software/parallel/parallel_tutorial.html
If you have not already done so, walk through the tutorial; your command line will love you for it: http://www.gnu.org/software/parallel/parallel_tutorial.html
这篇关于在管道中使用 GNU 并行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!