读取Bash管道中的块中的标准输入 [英] Read stdin in chunks in Bash pipe

问题描述

我有一些shell脚本可用于类似这样的管道:

I have some shell scripts that works with pipes like such:

foo.sh | bar.sh

我的bar.sh调用某些命令行程序，该程序只能占用一定数量的stdin行.因此，我希望foo.sh的大stdout在N行中被分块以进行多个bar.sh调用.本质上，分页foo.sh的标准输出并执行多个bar.sh.

My bar.sh calls some command line program that can only take a certain number of lines of stdin. Thus, I want foo.sh's large stdout to be chunked up in N number of lines to make multiple bar.sh calls. Essentially, paginate foo.sh's stdout and do multiple bar.sh.

有可能吗?我希望像foo.sh | ??? | bar.sh这样的管道之间有一些魔力. xargs -n并不能完全满足我的需求.

Is it possible? I am hoping for some magic in between the pipes like foo.sh | ??? | bar.sh. xargs -n doesn't quite get me what I want.

推荐答案

我距离测试该机器不远，但是您需要GNU Parallel使其变得容易-遵循以下原则:

I am nowhere near a machine to test this, but you need GNU Parallel to make this easy - along the lines of:

foo.sh | parallel --pipe -N 10000 -k bar.sh

作为一个额外的好处，它将与您拥有CPU内核的并行运行.

As an added bonus, that will run as many bar.sh in parallel as you have CPU cores.

如果一次只需要一个bar.sh，则添加-j 1.

Add -j 1 if you only want one bar.sh at a time.

如果要查看它将执行什么操作但不执行任何操作，请添加--dry-run.

Add --dry-run if you want to see what it would do but without doinng anything.

这篇关于读取Bash管道中的块中的标准输入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！