即时输出重定向，在程序仍在运行时查看文件重定向输出 [英] on-the-fly output redirection, seeing the file redirection output while the program is still running

问题描述

如果我使用这样的命令:
./program >> a.txt&
，并且该程序运行时间很长，所以一旦程序结束，我只能看到输出.这意味着我无法知道计算是否顺利进行，直到实际停止计算为止.我希望能够在程序运行时读取文件上的重定向输出.

If I use a command like this one:
./program >> a.txt &
, and the program is a long running one then I can only see the output once the program ended. That means I have no way of knowing if the computation is going well until it actually stops computing. I want to be able to read the redirected output on file while the program is running.

这类似于打开文件，将其附加到文件中，然后在每次写入后将其关闭.如果仅在程序结束时关闭文件，则在程序结束之前无法在其上读取任何数据.我知道的唯一重定向类似于在程序结束时关闭文件.

This is similar to opening a file, appending to it, then closing it back after every writing. If the file is only closed at the end of the program then no data can be read on it until the program ends. The only redirection I know is similar to closing the file at the end of the program.

您可以使用此小python脚本对其进行测试.语言无关紧要.任何写入标准输出的程序都存在相同的问题.

You can test it with this little python script. The language doesn't matter. Any program that writes to standard output has the same problem.

l = range(0,100000)
for i in l:
  if i%1000==0:
    print i
  for j in l:
    s = i + j

一个人可以用以下命令运行它:
./python program.py >> a.txt&
然后cat a.txt ..您只有在脚本完成计算后才能得到结果.

One can run this with:
./python program.py >> a.txt &
Then cat a.txt .. you will only get results once the script is done computing.

推荐答案

来自 stdout手册页:

stderr流没有缓冲. 流标准输出是行缓冲的 当它指向终点站时. 直到出现分行 fflush(3)或exit(3)被调用，或 换行.

The stream stderr is unbuffered. The stream stdout is line-buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a new‐line is printed.

底线:除非输出是终端，否则默认情况下，程序的标准输出将处于完全缓冲模式.从本质上讲，这意味着它将以大块代码输出数据，而不是逐行输出数据，更不用说逐个字符了.

Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.

解决此问题的方法:

• 修复程序:如果需要实时输出，则需要修复程序.在C语言中，您可以在每个输出语句后使用 fflush(stdout) 或 setvbuf() 来更改标准输出的缓冲模式.对于Python，甚至在此处.

使用可以从PTY记录的实用程序，而不是直接执行stdout重定向. GNU屏幕可以为您做到这一点:

Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screen can do this for you:

screen -d -m -L python test.py

将是一个开始.这会将程序的输出记录到当前目录中名为screenlog.0(或类似名称)的文件中，默认延迟为10秒，并且您可以使用screen连接到运行命令的会话以提供输入或终止它.延迟和日志文件的名称可以在配置文件中更改，也可以在连接到后台会话后手动更改.

would be a start. This will log the output of your program to a file called screenlog.0 (or similar) in your current directory with a default delay of 10 seconds, and you can use screen to connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.

在大多数Linux系统上，存在第三个解决方法:您可以使用LD_PRELOAD变量和预加载的库来覆盖C库的选择函数，并在调用以下函数时使用它们来设置stdout缓冲模式.您的程序.此方法可能有效，但有很多缺点:

On most Linux system there is a third workaround: You can use the LD_PRELOAD variable and a preloaded library to override select functions of the C library and use them to set the stdout buffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:

• 在静态可执行文件上根本无法使用

• It won't work at all on static executables

它很脆弱而且很丑.

对于SUID可执行文件根本不起作用-出于安全原因，动态加载程序在加载此类可执行文件时将拒绝读取LD_PRELOAD变量.

It won't work at all with SUID executables - the dynamic loader will refuse to read the LD_PRELOAD variable when loading such executables for security reasons.

它很脆弱而且很丑.

它要求您找到并覆盖程序之后调用的库函数，它最初设置了stdout缓冲模式，最好设置为 before 输出. getenv()是许多程序(但不是全部)的不错选择.您可能必须重写常见的I/O功能，例如printf()或fwrite()-如果一推再推，您可能只需要重写所有控制缓冲模式的功能，并为stdout引入特殊条件即可.

It requires that you find and override a library function that is called by your program after it initially sets the stdout buffering mode and preferably before any output. getenv() is a good choice for many programs, but not all. You may have to override common I/O functions such as printf() or fwrite() - if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition for stdout.

它很脆弱而且很丑.

很难确保没有不受欢迎的副作用.要正确执行此操作，您必须确保仅影响stdout，并且如果例如stdout已关闭.

It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only stdout is affected and that your overrides will not crash the rest of the program if e.g. stdout is closed.

我是否提到了它的脆弱和丑陋?

Did I mention that it's fragile and rather ugly?

也就是说，过程相对简单.您放入一个C文件，例如linebufferedstdout.c替换功能:

That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.c the replacement functions:

#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>


char *getenv(const char *s) {
    static char *(*getenv_real)(const char *s) = NULL;

    if (getenv_real == NULL) {
        getenv_real = dlsym(RTLD_NEXT, "getenv");

        setlinebuf(stdout);
    }

    return getenv_real(s);
}

然后将文件编译为共享对象:

Then you compile that file as a shared object:

gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc

然后设置LD_PRELOAD变量以将其与程序一起加载:

Then you set the LD_PRELOAD variable to load it along with your program:

$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out 
0
1000
2000
3000
4000

如果幸运的话，您的问题将得到解决，而不会出现不幸的副作用.

If you are lucky, your problem will be solved with no unfortunate side-effects.

如有必要，您可以在外壳中设置LD_PRELOAD库，甚至可以在/etc/ld.so.preload中指定系统范围的库(建议绝对 NOT ).

You can set the LD_PRELOAD library in the shell, if necessary, or even specify that library system-wide (definitely NOT recommended) in /etc/ld.so.preload.

这篇关于即时输出重定向，在程序仍在运行时查看文件重定向输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！