如何从多个进程中拆分并重新加入STDOUT? [英] How can I split and re-join STDOUT from multiple processes?

问题描述

我正在处理一个带有几个分支点并随后合并的管道-它们看起来像这样:

I am working on a pipeline that has a few branch points that subsequently merge-- they look something like this:

         command2
        /        \
command1          command4
        \        /
         command3

每个命令写入STDOUT并接受通过STDIN的输入.来自command1的STDOUT必须同时传递到 command2和command3，它们依次运行 ，并且它们的输出必须有效地串联并传递给command4.我最初以为这样的事情会起作用:

Each command writes to STDOUT and accepts input via STDIN. STDOUT from command1 needs to be passed to both command2 and command3, which are run sequentially, and their output needs to be effectively concatenated and passed to command4. I initially thought that something like this would work:

$ command1 | (command2; command3) | command4

但这不起作用，因为仅将command2的STDOUT传递给命令4，并且当我删除command4时，很明显，command3没有从command1传递适当的流-换句话说，就好像command2正在耗尽或消耗流.我得到与{command2;相同的结果command3; }也在中间.所以我想我应该使用，并尝试这样做:

That doesn't work though, as only STDOUT from command2 is passed to command 4, and when I remove command4 it's apparent that command3 isn't being passed the appropriate stream from command1 -- in other words, it's as if command2 is exhausting or consuming the stream. I get the same result with { command2 ; command3 ; } in the middle as well. So I figured I should be using 'tee' with process substitution, and tried this:

$ command1 | tee >(command2) | command3 | command4

但是令人惊讶的是，它也没有起作用-看来command1的输出和的命令2的输出通过管道传递到command3，这会导致错误，并且只有command3的输出通过管道传递到命令4.我确实发现以下内容可以从command2和command3处获取适当的输入和输出:

But surprisingly that didn't work either -- it appears that the output of command1 and the output of command2 are piped into command3, which results in errors and only the output of command3 being piped into command4. I did find that the following gets the appropriate input and output to and from command2 and command3:

$ command1 | tee >(command2) >(command3) | command4

但是，这也会将command1的输出流式传输到command4，这会导致问题，因为command2和command3产生的规范与command1不同.我已经找到的解决方案似乎很棘手，但它确实有效:

However, this streams the output of command1 to command4 as well, which leads to issues as command2 and command3 produce a different specification than command1. The solution I've arrived on seems hacky, but it does work:

$ command1 | tee >(command2) >(command3) > /dev/null | command4

在从command2和command3收集STDOUT的同时，禁止command1将其输出传递给command4.它可以工作，但是我感觉缺少一个更明显的解决方案.是吗我已经阅读了数十个线程，还没有找到适用于我的用例的解决方案，也没有看到关于拆分和重新加入流的确切问题的详细说明(尽管我不能成为第一个)一个来解决这个问题).我应该只使用命名管道吗?我尝试过，但是也很难使它正常工作，所以也许这是另一个话题的另一个故事.我在RHEL5.8中使用bash.

That suppresses command1 passing its output to command4, while collecting STDOUT from command2 and command3. It works, but I feel like I'm missing a more obvious solution. Am I? I've read dozens of threads and haven't found a solution to this problem that works in my use case, nor have I seen an elaboration of the exact problem of splitting and re-joining streams (though I can't be the first one to deal with this). Should I just be using named pipes? I tried but had difficulty getting that working as well, so maybe that's another story for another thread. I'm using bash in RHEL5.8.

推荐答案

您可以像这样使用文件描述符；

You can play around with file descriptors like this;

((date | tee >( wc >&3) | wc) 3>&1) | wc

或

((command1 | tee >( command2 >&3) | command3) 3>&1) | command4

为了说明，即tee >( wc >&3)将在stdout上输出原始数据，而内部的wc将在FD 3上输出结果.外部3>& 1)然后将FD3的输出合并回STDOUT，因此两个wc的输出都发送到拖尾命令.

To explain, that is tee >( wc >&3) will output the original data on stdout, and the inner wc will output the result on FD 3. The outer 3>&1) will then merge FD3 output back into STDOUT so output from both wc is sent to the tailing command.

但是，此管道中没有任何内容(或您自己的解决方案中的任何内容)可以保证不会损坏输出.那就是来自command2的不完整行不会与command3的行混在一起-如果这是一个问题，您将需要执行以下两项操作之一:

HOWEVER, there is nothing in this pipeline (or the one in your own solution) which will guanrantee that the output will not be mangled. That is incomplete lines from command2 will not be mixed up with lines of command3 -- if that is a concern, you will need to do one of two things;

1. 编写您自己的tee程序，该程序内部使用popen并读回每一行，然后再将完整的行发送到stdout以供command4读取
2. 将command2和command3的输出写入文件，并使用cat将数据合并为command4的输入

1. Write your own tee program which internally uses popen and read each line back before sending complete lines to stdout for command4 to read
2. Write the output from command2 and command3 to a file and use cat to merge the data as input to command4

这篇关于如何从多个进程中拆分并重新加入STDOUT?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！