将STDOUT和STDERR重定向到python logger以及jupyter Notebook [英] Redirect STDOUT and STDERR to python logger and also to jupyter notebook

问题描述

重要信息:我正在使用jupyter笔记本.

我想创建一个记录器，将STDOUT和STDERR重定向到该记录器，但我也想在jupyter笔记本输出控制台上看到这些输出.

I want to create a logger to which I will redirect the STDOUT and STDERR but I also want to see those outputs on the jupyter notebook output console.

到目前为止，我已经实现的是:

So far what I have implemented is:

import logging
import sys

class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, log_level=logging.INFO):
        self.logger = logger
        self.log_level = log_level
        self.linebuf = ''

    def write(self, buf):
        for line in buf.rstrip().splitlines():
            self.logger.log(self.log_level, line.rstrip())

    def flush(self):
        pass

logging.basicConfig(filename='my_log.log',
                    filemode='a',
                    # stream=sys.stdout,
                    level=logging.DEBUG,
                    format='%(asctime)s;%(name)s;%(levelname)s;%(message)s')

# redirect stdout and stderr to logger
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl

stderr_logger = logging.getLogger('STDERR')
s2 = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = s2

log = logging.getLogger('my_log')

# An info log
log.info('This is an info')

# A stdout message
print("This is an STDOUT")

# A stderr message
1 / 0

第一个问题: 如果先前的代码存储在 .py 文件中，则可以将 stdout 和 stderr 重定向到 my_log.log 文件.但是我无法在普通的终端控制台中跟踪消息.

First Question: The previous code, if stored on a .py file, is able to redirect the stdout and stderr to the my_log.log file. But I lose track of the messages in a normal terminal console.

我想将 stderr 和 stdout 都重定向到日志文件，并且还可以在控制台上看到它们.

What I would like is to have both the stderr and stdout redirected to the log file and also be able to see them on the console.

第二个问题: 我正在研究Jupyter笔记本，我希望能够从那里登录.这意味着所有的stdout和stderr从jupyter笔记本输出重定向到日志文件，但也将其保留在jupyter笔记本控制台上. 我意识到上面的代码将stdout重定向到日志文件，而不是stderr，结果我的所有打印文件('XX')都保存在我的日志文件中，而我的异常仍然在笔记本控制台上.

Second Question: I am working on jupyter notebooks and I would like to be able to log from there. This means all the stdout and stderr redirected from jupyter notebook output to the log file, but also keep it on the jupyter notebook console. I realized that the code above, redirects the stdout to the log file but not the stderr, and as a result all my prints('XX') are in my log file and my exceptions are still on the notebook console.

看起来像jupyter笔记本电脑与STDOUT和STDERR的交易方式不同

Seems like jupyter notebooks deals in a different way with STDOUT and STDERR

感谢您的帮助

推荐答案

可能会晚一些.
我有类似的要求，而找到最佳方法是与以下代码类似的东西.我用它来跟踪将持续数天的模型训练的输出.它可以通过调用Std2File.enable(file_name)登录到文件，并可以通过调用Std2File.disable()进行回滚. 它可以登录到文件并显示到笔记本中.唯一的不足是笔记本电脑上显示的输出将始终显示在调用enable的单元格中.

It might be a little late.
I had similar demands and the best way I can find to do it is something similar to the following code. I use it to track outputs of model training that will run for days. It can log to a file by calling Std2File.enable(file_name) and rollback by calling Std2File.disable(). It can log to the file and show it to the notebook. the only shortness is that the output displayed on the notebook will always show in the cell of calling enable.

class Std2File(object):
    """
    Redirect stoout and stderr to a local file.
    It is designed for use case of notebook where after you close a notebook you can record the stdout and stderr
    """
    stdout = None
    stderr = None
    fd = None

    def __init__(self, f, std):
        self.std = std
        self.f = f

    @staticmethod
    def enable(f='/tmp/std_copy.log'):
        if Std2File.stdout is None:
            Std2File.stdout = sys.stdout
            Std2File.stderr = sys.stderr
            Std2File.fd = open(f, 'a+')
            sys.stdout = Std2File(Std2File.fd, sys.stdout)
            sys.stderr = Std2File(Std2File.fd, sys.stderr)
        print('\n\nbackup stdout/stderr to %s at %s\n' % (f, dt.datetime.now()))

    @staticmethod
    def disable():
        if Std2File.stdout is not None:
            sys.stdout = Std2File.stdout
            sys.stderr = Std2File.stderr
            Std2File.stderr = Std2File.stdout = None
            Std2File.fd.close()
            Std2File.fd = None

    def __getattr__(self, name):
        return object.__getattribute__(self.f, name)

    def write(self, x):
        self.std.write(x)
        self.std.flush()
        self.f.write(x)
        self.f.flush()

    def flush(self):
        self.std.flush()
        self.f.flush()

这篇关于将STDOUT和STDERR重定向到python logger以及jupyter Notebook的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！