将几个子流程整合在一起 [英] piping together several subprocesses

问题描述

我有5个进程p1,p2,...,p5，我想将一些数据写入p1的stdin，将p1输出到p2 stdin，最后从p5的输出中读取最终结果.

I have 5 processes p1,p2,...,p5 where I want to write some data to stdin of p1, pipe p1 output to p2 stdin and finally read the final result from output of p5.

到目前为止我尝试过的事情:

What I have tried so far:

p1 = Popen(['p1'], stdin=PIPE, stdout=PIPE)
p2 = Popen(['p2'], stdin=p1.stdout, stdout=PIPE)
...
p5 = Popen(['p5'], stdin=p4.stdout, stdout=PIPE)

# write data to stdin
p1.stdin.write(indata)
p1.stdin.close()

# not sure in what order to close the pipes here, if at all

# read output
out = p5.stdout.read()
print out

最后一个被截断的代码只是挂起，因为我必须不正确地执行读/写操作.

The last code snipped just hangs, because I must be doing the read/write operations incorrectly.

我能够使用communicate()和两个进程工作，而无需向第一个进程提供任何输入(例如来自Python文档的示例):

I was able to get working single processes using communicate() and two processes without suppling any input to first one (example from Python docs):

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

但是我不知道如何在不停止解释器的情况下为第一过程提供输入.

But I cannot figure out, how to supply input to first process without hanging the interpreter.

我也可以为此使用bash脚本(我已经编写并可以使用它)，但是我想知道如何使用Python实现相同的功能.

I could also use a bash script for that (that I've already written and works), but I'd like to know, how to achieve same with Python.

所以，我想问一下，如何正确地完成所有操作，特别是按什么顺序对管道进行读/写/关闭操作?

So, I'd like to ask, how do do all that properly, specifically in what order to do the read/write/close operations on pipes?

如果这很重要，我正在使用64位Linux.

I'm working on 64-bit Linux, if that matters.

我忘了提及所有进程p1，.. p5消耗它们给定的所有输入，对其进行处理，然后写入stdout，然后终止.因此，流水线中的下一个进程不应在前一个进程完成处理之前终止.

I forgot to mention that all processes p1,..p5 consume all input they are given, process it, write to stdout and then terminate. Thus, the processes next in pipeline should not terminate before the previous ones have finished processing.

我知道我也可以使用

command = 'bash -c "p1 | p2 | p3 | p4 | p5"'
proc = Popen([command], shell=True)
out, err = proc.communicate(input=indata)
print out

但是我的主要兴趣是了解如何纯粹用python代码链接管道.

but my main interest is getting know how to chain the pipes purely in python code.

推荐答案

也许可以帮忙:

import sys
import tempfile
from subprocess import Popen, PIPE


cmd = [sys.executable, '-c', 'print raw_input()']

# Using a temp file to give input data to the subprocess instead of stdin.write to avoid deadlocks.
with tempfile.TemporaryFile() as f:
    f.write('foobar')
    f.seek(0)  # Return at the start of the file so that the subprocess p1 can read what we wrote.
    p1 = Popen(cmd, stdin=f, stdout=PIPE)

p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE)
p3 = Popen(cmd, stdin=p2.stdout, stdout=PIPE)

# No order needed.
p1.stdout.close()
p2.stdout.close()

# Using communicate() instead of stdout.read to avoid deadlocks. 
print p3.communicate()[0]

输出:

$ python test.py
foobar

希望这会很幸福.

这篇关于将几个子流程整合在一起的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！