将子流程的stdout重定向到2个或更多子流程的stdin [英] Redirect stdout of a subprocess to stdin of 2 or more subprocesses
问题描述
基本上,我想学习如何使用一个subprocess
(例如proc1
)的stdout
作为2个或更多其他subprocess
es(例如proc2
& proc3
)的stdin
的用法.在python中.
Basically I want to learn how to use the stdout
of one subprocess
(say proc1
) as stdin
of 2 or more other subprocess
es (say proc2
& proc3
) in python.
我需要zcat
.gz文件,并将发送到subprocess.PIPE
的输出用于cksum
(unix实用程序)和行数.
Hi,
I need to zcat
a .gz file and use the output sent to subprocess.PIPE
for both cksum
(unix utility) and to line count.
我可以像这样用bash来做...
I can do it in bash like this...
[hashroot@dev_server 12]$ zcat ABC_C_TPM_26122014.data.gz | tee >(wc -l) >(cksum)| tail -2
2020090579 112180
586
我想在python中做同样的事情.
I want to do the same in python.
我一旦这样做...
>>> import subprocess
>>> import os
>>> fl123 = 'ABC_C_TPM_26122014.data.gz'
>>> pqr123 = subprocess.Popen(['zcat', fl123], stdout=subprocess.PIPE)
>>> subprocess.check_output(['cksum'], stdin=pqr123.stdout)
b'4286000649 256100 \n'
现在PIPE
为空,所以如何获得行数,直到不再执行zcat
.
Now the PIPE
is empty so how will I get line count till I don't do zcat
again.
通过在子进程中运行zcat两次,然后将第一个zcat
输出重定向到wc -l并将第二个zcat
输出重定向到cksum
,我可以很好地做到这一点.但是zcat
是基于磁盘IO的,速度很慢.所以我想避免它.
I can very well do it by running zcat twice in subprocess, and redirecting the first zcat
output to wc -l and the second zcat
's output to cksum
. But zcat
is disk IO based and is slow. So I want to avoid it.
推荐答案
在Python中实现tee
命令的一种简单方法是手动写入子过程:
A simple way to implement the tee
command in Python is to write to the subprocesses manually:
import gzip
from subprocess import Popen, PIPE
# zcat ABC_C_TPM_26122014.data.gz | tee >(wc -l) >(cksum)
with gzip.open("ABC_C_TPM_26122014.data.gz", "rb") as input_file:
wc = Popen(['wc', '-l'], stdin=PIPE, bufsize=1, close_fds=True)
cksum = Popen(['cksum'], stdin=PIPE, bufsize=1, close_fds=True)
line_count = 0
for line_count, line in enumerate(input_file, start=1):
wc.stdin.write(line)
cksum.stdin.write(line)
wc.stdin.close()
cksum.stdin.close()
wc.wait()
cksum.wait()
print("Line count in the parent: %d" % line_count)
如果输入中的行可能很大,那么您可以分块读取输入:chunk = input_file.read(chunk_size)
而不是逐行(b'\n'
).
If the lines in the input can be large then you could read the input in chunks: chunk = input_file.read(chunk_size)
instead of line by line (b'\n'
).
这篇关于将子流程的stdout重定向到2个或更多子流程的stdin的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!