在Python中使用多重处理时应如何记录? [英] How should I log while using multiprocessing in Python?

问题描述

现在，我在框架中有一个中央模块，该模块使用Python 2.6产生了多个进程.文档，此记录器具有进程共享锁，因此您不必通过使多个进程同时写入sys.stderr(或任何文件句柄)，可以弄乱东西.

Right now I have a central module in a framework that spawns multiple processes using the Python 2.6 multiprocessing module. Because it uses multiprocessing, there is module-level multiprocessing-aware log, LOG = multiprocessing.get_logger(). Per the docs, this logger has process-shared locks so that you don't garble things up in sys.stderr (or whatever filehandle) by having multiple processes writing to it simultaneously.

我现在遇到的问题是框架中的其他模块不支持多处理.以我的方式看，我需要使这个中央模块上的所有依赖项都使用支持多处理的日志记录.在框架内 令人讨厌，更不用说框架的所有客户了.有我没有想到的替代方法吗?

The issue I have now is that the other modules in the framework are not multiprocessing-aware. The way I see it, I need to make all dependencies on this central module use multiprocessing-aware logging. That's annoying within the framework, let alone for all clients of the framework. Are there alternatives I'm not thinking of?

推荐答案

唯一地解决此问题的方法是:

The only way to deal with this non-intrusively is to:

1. 生成每个工作进程，以使其日志转到不同的文件描述符(到磁盘或管道).理想情况下，应为所有日志条目加上时间戳.
2. 您的控制器进程随后可以执行以下一项操作:
   • 如果使用磁盘文件::在运行结束时合并日志文件(按时间戳排序)
   • 如果使用管道(推荐):将所有管道动态地将日志条目合并到中央日志文件中. (例如，定期从管道的文件描述符中 select ，对可用的日志条目，然后刷新到集中式日志.重复.)

1. Spawn each worker process such that its log goes to a different file descriptor (to disk or to pipe.) Ideally, all log entries should be timestamped.
2. Your controller process can then do one of the following:
   • If using disk files: Coalesce the log files at the end of the run, sorted by timestamp
   • If using pipes (recommended): Coalesce log entries on-the-fly from all pipes, into a central log file. (E.g., Periodically select from the pipes' file descriptors, perform merge-sort on the available log entries, and flush to centralized log. Repeat.)

这篇关于在Python中使用多重处理时应如何记录?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！