如何从 Python 迭代器提供子流程的标准输入? [英] How can I feed a subprocess's standard input from a Python iterator?

问题描述

我正在尝试使用 Python 中的 subprocess 模块与以流式方式读取标准输入和写入标准输出的进程进行通信.我想让子进程从产生输入的迭代器中读取行，然后从子进程中读取输出行.输入和输出线之间可能没有一一对应.如何从返回字符串的任意迭代器提供子进程?

I am trying to use the subprocess module in Python to communicate with a process that reads standard input and writes standard output in a streaming fashion. I want to have the subprocess read lines from an iterator that produces the input, and then read output lines from the subprocess. There may not be a one-to-one correspondence between input and output lines. How can I feed a subprocess from an arbitrary iterator that returns strings?

以下是一些示例代码，提供了一个简单的测试用例，以及我尝试过的一些方法，但由于某种原因或其他原因不起作用:

Here is some example code that gives a simple test case, and some methods I have tried that don't work for some reason or other:

#!/usr/bin/python
from subprocess import *
# A really big iterator
input_iterator = ("hello %s
" % x for x in xrange(100000000))

# I thought that stdin could be any iterable, but it actually wants a
# filehandle, so this fails with an error.
subproc = Popen("cat", stdin=input_iterator, stdout=PIPE)

# This works, but it first sends *all* the input at once, then returns
# *all* the output as a string, rather than giving me an iterator over
# the output. This uses up all my memory, because the input is several
# hundred million lines.
subproc = Popen("cat", stdin=PIPE, stdout=PIPE)
output, error = subproc.communicate("".join(input_iterator))
output_lines = output.split("
")

那么，当我逐行从其标准输出中读取时，如何让我的子进程逐行从迭代器中读取?

So how can I have my subprocess read from an iterator line by line while I read from its stdout line by line?

推荐答案

最简单的方法似乎是从子进程 fork 并提供输入句柄.任何人都可以详细说明这样做的任何可能的缺点吗?或者有没有让它更容易和更安全的 Python 模块?

The easy way seems to be to fork and feed the input handle from the child process. Can anyone elaborate on any possible downsides of doing this? Or are there python modules that make it easier and safer?

#!/usr/bin/python
from subprocess import *
import os

def fork_and_input(input, handle):
    """Send input to handle in a child process."""
    # Make sure input is iterable before forking
    input = iter(input)
    if os.fork():
        # Parent
        handle.close()
    else:
        # Child
        try:
            handle.writelines(input)
            handle.close()
        # An IOError here means some *other* part of the program
        # crashed, so don't complain here.
        except IOError:
            pass
        os._exit()

# A really big iterator
input_iterator = ("hello %s
" % x for x in xrange(100000000))

subproc = Popen("cat", stdin=PIPE, stdout=PIPE)
fork_and_input(input_iterator, subproc.stdin)

for line in subproc.stdout:
    print line,

这篇关于如何从 Python 迭代器提供子流程的标准输入?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！