Python:select()并不表示管道中的所有输入 [英] Python: select() doesn't signal all input from pipe

问题描述

我试图用Python加载外部命令行程序并通过管道与之通信.该程序将通过stdin输入文本，并以行的形式输出文本到stdout.使用select()进行通信应该是异步的.

I am trying to load an external command line program with Python and communicate with it via pipes. The progam takes text input via stdin and produces text output in lines to stdout. Communication should be asynchronous using select().

问题是，并非程序的所有输出都在select()中发出信号.通常，最后一两行不发信号.如果select()返回超时，并且我试图从管道中读取，则readline()立即返回程序发送的行.参见下面的代码.

The problem is, that not all output of the program is signalled in select(). Usually the last one or two lines are not signalled. If select() returns with a timeout and I am trying to read from the pipe anyway readline() returns immediately with the line sent from the program. See code below.

程序不会缓冲输出，而是以文本行发送所有输出.到目前为止，通过许多其他语言和环境的管道连接到程序的效果很好.

The program doesn't buffer the output and sends all output in text lines. Connecting to the program via pipes in many other languages and environments has worked fine so far.

我已经在Mac OSX 10.6上尝试了Python 3.1和3.2.

I have tried Python 3.1 and 3.2 on Mac OSX 10.6.

import subprocess
import select

engine = subprocess.Popen("Engine", bufsize=0, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
engine.stdin.write(b"go\n")
engine.stdin.flush()

while True:
    inputready,outputready,exceptready = select.select( [engine.stdout.fileno()] , [], [], 10.0)

    if (inputready, outputready, exceptready) == ([], [], []):
        print("trying to read from engine anyway...")
        line = engine.stdout.readline()
        print(line)

     for s in inputready:
        line = engine.stdout.readline()
        print(line)

推荐答案

请注意，内部file.readlines([size])循环并多次调用read() syscall，试图填充size的内部缓冲区.对read()的第一次调用将立即返回，因为select()表示fd是可读的.但是，第二个调用将阻塞，直到有可用数据为止，这违背了使用select的目的.无论如何，在异步应用程序中使用file.readlines([size])都是棘手的.

Note that internally file.readlines([size]) loops and invokes the read() syscall more than once, attempting to fill an internal buffer of size. The first call to read() will immediately return, since select() indicated the fd was readable. However the 2nd call will block until data is available, which defeats the purpose of using select. In any case it is tricky to use file.readlines([size]) in an asynchronous app.

对于每一次通过选择，都应在每个fd上调用一次os.read(fd, size).这将执行无阻塞读取，并允许您缓冲部分行，直到数据可用并明确检测到EOF.

You should call os.read(fd, size) once on each fd for every pass through select. This performs a non-blocking read, and lets you buffer partial lines until data is available and detects EOF unambiguously.

我修改了您的代码以说明如何使用os.read.它还从进程的stderr中读取:

I modified your code to illustrate using os.read. It also reads from the process' stderr:

import os
import select
import subprocess
from cStringIO import StringIO

target = 'Engine'
PIPE = subprocess.PIPE
engine = subprocess.Popen(target, bufsize=0, stdin=PIPE, stdout=PIPE, stderr=PIPE)
engine.stdin.write(b"go\n")
engine.stdin.flush()

class LineReader(object):

    def __init__(self, fd):
        self._fd = fd
        self._buf = ''

    def fileno(self):
        return self._fd

    def readlines(self):
        data = os.read(self._fd, 4096)
        if not data:
            # EOF
            return None
        self._buf += data
        if '\n' not in data:
            return []
        tmp = self._buf.split('\n')
        lines, self._buf = tmp[:-1], tmp[-1]
        return lines

proc_stdout = LineReader(engine.stdout.fileno())
proc_stderr = LineReader(engine.stderr.fileno())
readable = [proc_stdout, proc_stderr]

while readable:
    ready = select.select(readable, [], [], 10.0)[0]
    if not ready:
        continue
    for stream in ready:
        lines = stream.readlines()
        if lines is None:
            # got EOF on this stream
            readable.remove(stream)
            continue
        for line in lines:
            print line

这篇关于Python:select()并不表示管道中的所有输入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！