Python 子进程 - 过滤掉日志记录 [英] Python Subprocess - filter out logging

问题描述

Python 3.6

我想从我使用 subprocess 模块运行的子进程中获取所有输入.我可以轻松地将此输出通过管道传输到日志文件，而且效果很好.

I want to take all input from a subprocess which I run with the subprocess module. I can easily pipe this output to a log file, and it works great.

但是，我想过滤掉很多行(来自我无法控制的模块的大量嘈杂输出).

But, I want to filter out a lot of the lines (lots of noisy output from modules I do not control).

def run_command(command, log_file):
    process = subprocess.Popen(command, stdout=subprocess.PIPE,
                               stderr=subprocess.STDOUT, bufsize=1,
                               universal_newlines=True)
    while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output and not_noisy_line(output):
            log_file.write(output)
            log_file.flush()
    return process.poll()

但这在我的子进程和输出之间引入了竞争条件.

But this introduced a race condition between my subprocess and the output.

我创建了一个新方法和一个类来包装日志记录.

I created a new method and a class to wrap the logging.

def run_command(command, log_file):
    process = subprocess.run(command, stdout=QuiteLogger(log_file), stderr=QuiteLogger(log_file), timeout=120)
    return process.returncode

class QuiteLogger(io.TextIOWrapper):
    def write(self, data, encoding=sys.getdefaultencoding()):
        data = filter(data)
        super().write(data)

然而，这只是完全跳过了我的过滤器功能，我的 write 方法根本没有被子进程调用.(如果我调用 QuietLogger().write('asdasdsa') 它会通过过滤器)

This does however just completely skip my filter function, my write method is not called at all by the subprocess. (If I call QuietLogger().write('asdasdsa') it goes through the filters)

有什么线索吗?

推荐答案

这是一个有趣的情况，其中文件对象抽象部分崩溃.您的解决方案不起作用的原因是 subprocess 实际上并未使用您的 QuietLogger 而是从中获取原始文件号(然后将其重新打包为 io.TextIOWrapper 对象).

This is an interesting situation in which the file object abstraction partially breaks down. The reason your solution does not work, is because subprocess is not actually using your QuietLogger but is getting the raw file number out of it (then repackaging it as a io.TextIOWrapper object).

我不知道这是否是 subprocess 处理方式的内在限制，依赖于操作系统支持，或者这是否只是 Python 设计中的一个错误，但为了实现你想要什么，你需要使用标准的 subprocess.PIPE 然后滚动你自己的文件编写器.

I don't know if this is an intrinsic limitation in how the subprocess is handled, relying on OS support, or if this is just a mistake in the Python design, but in order to achieve what you want, you need to use the standard subprocess.PIPE and then roll your own file writer.

如果你可以等待子进程完成，那么它可以很容易地完成，使用 subprocess.run 然后从 CompletedProcess (<代码>p) 对象:

If you can wait for the subprocess to finish, then it can be trivially done, using the subprocess.run and then picking the stream out of the CompletedProcess (p) object:

p = subprocess.run(command, stdout=subprocess.PIPE, universal_newlines=True)

data = filter(p.stdout)

with open(logfile, 'w') as f:
    f.write(data)

如果您必须在生成时处理输出(因此，您不能等待 subprocess 结束)，最简单的方法是求助于 subprocess.Popen 和线程:

If you must work with the ouput while it is being generated (thus, you cannot wait for the subprocess to end) the simplest way is to resort to subprocess.Popen and threads:

import subprocess
import threading

logfile ='tmp.txt'
filter_passed = lambda line: line[:3] != 'Bad'
command = ['my_cmd', 'arg']

def writer(p, logfile):
    with open(logfile, 'w') as f:
        for line in p.stdout:
            if filter_passed(line):
                f.write(line)

p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)

t = threading.Thread(target=writer, args=(p,logfile))
t.start()

t.join()

这篇关于Python 子进程 - 过滤掉日志记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！