绕过子输出缓冲在C或Python的popen [英] Bypassing buffering of subprocess output with popen in C or Python

问题描述

我有关于popen这一个一般性的问题（和所有相关的功能），适用于所有操作系统，当我写一个python脚本或一些C code和从控制台运行生成的可执行文件（赢或Linux）我可以马上看到从过程的输出。但是，如果我运行相同的可执行文件与标准输出一个分叉过程重定向到一个管道，输出缓冲器的地方，一般可达4096字节之前它被写入到管道所在的父进程可以读取它。

I have a general question about popen (and all related functions), applicable to all operating systems, when I write a python script or some c code and run the resulting executable from the console (win or linux), i can immediately see the output from the process. However, if I run the same executable as a forked process with its stdout redirected into a pipe, the output buffers somewhere, usually up to 4096 bytes before it is written to the pipe where the parent process can read it.

下面的python脚本将产生1024字节的块输出

The following python script will generate output in chunks of 1024 bytes

import os, sys, time

if __name__ == "__main__":
     dye = '@'*1024
     for i in range (0,8):
        print dye
        time.sleep(1)

下面的python脚本将执行previous脚本，并尽快读取输出，因为它涉及到管道，逐字节

The following python script will execute the previous script and read the output as soon as it comes to the pipe, byte by byte

import os, sys, subprocess, time, thread

if __name__ == "__main__":
    execArgs = ["c:\\python25\\python.exe", "C:\\Scripts\\PythonScratch\\byte_stream.py"]

    p = subprocess.Popen(execArgs, bufsize=0, stdout=subprocess.PIPE)
    while p.returncode == None:
        data = p.stdout.read(1)
        sys.stdout.write(data)
        p.poll()

调整您的操作系统的路径。当在此配置中运行，输出将不会，出现在1024块，但4096块尽管器一起命令的缓冲器大小被设置为0（这是默认反正）。谁能告诉我如何改变这种现象？有没有什么办法可以强制操作系统来治疗从叉过程的输出以同样的方式，当它从控制台运行的?,即刚刚经历填充数据没有缓冲？

Adjust the path for your operating system. When run in this configuration, the output will not appear in chunks of 1024 but chunks of 4096, despite the buffer size of the popen command being set to 0 (which is the default anyway). Can anyone tell me how to change this behaviour?, is there any way I can force the operating system to treat the output from the forked process in the same way as when it is run from the console?, ie, just feed the data through without buffering?

推荐答案

在一般情况下，标准的C运行时库（即每次的系统上运行的代表几乎所有的方案，或多或少;-)检测标准输出是否是终端与否;如果没有，它缓冲的输出（其可以是一个巨大的效率取胜，相比无缓冲输出）。

In general, the standard C runtime library (that's running on behalf of just about every program on every system, more or less;-) detects whether stdout is a terminal or not; if not, it buffers the output (which can be a huge efficiency win, compared to unbuffered output).

如果您在做这写程序的控制权的时候，你可以（作为另一个答案的建议）齐平标准输出连续，或（更优雅如果可行的话），试图迫使标准输出是无缓冲，例如通过与 -u 命令行标志运行Python的：

If you're in control of the program that's doing the writing, you can (as another answer suggested) flush stdout continuously, or (more elegantly if feasible) try to force stdout to be unbuffered, e.g. by running Python with the -u commandline flag:

-u     : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
         see man page for details on internal buffering relating to '-u'

（什么手册页添加为标准输入和二进制模式问题的提及[S]）。

(what the man page adds is a mention of stdin and issues with binary mode[s]).

如果您不能或不想触摸的编写程序， -u 或者这只是读取程序之类无助于（该缓冲才是最重要是一个发生在作家的标准输出，而不是一个在读者的标准输入）。另一种方法是作家欺骗，以为它写入终端（即使在事实上它写到另一个程序！），通过 PTY 标准库模块或更高-level第三方 Pexpect的模块（或者，对于Windows，它的端口的wexpect ）。

If you can't or don't want to touch the program that's writing, -u or the like on the program that's just reading is unlikely to help (the buffering that matters most is the one happening on the writer's stdout, not the one on the reader's stdin). The alternative is to trick the writer into believing that it's writing to a terminal (even though in fact it's writing to another program!), via the pty standard library module or the higher-level third party pexpect module (or, for Windows, its port wexpect).

这篇关于绕过子输出缓冲在C或Python的popen的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！