python多处理日志记录:带有RotatingFileHandler“正在由另一个进程使用的文件"的QueueHandler；错误 [英] python multiprocessing logging: QueueHandler with RotatingFileHandler "file being used by another process" error

问题描述

我正在将程序转换为多处理程序，并且需要能够从主流程以及子流程记录到单个循环日志中.我正在尝试在python食谱从多个进程登录到一个文件，这会启动logger_thread作为主进程的一部分运行，从子进程添加到的队列中拾取日志消息.该示例按原样运行，并且如果我切换到RotatingFileHandler，也可以运行.

但是，如果我将其更改为在子进程之前启动logger_thread(以便我也可以从主进程进行日志记录)，则一旦日志旋转，所有后续日志记录都会使用WindowsError: [Error 32] The process cannot access the file because it is being used by another process进行追溯. /p>

换句话说，我从第二个示例中更改了此代码

workers = []
for i in range(5):
    wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
    workers.append(wp)
    wp.start()
logging.config.dictConfig(d)
lp = threading.Thread(target=logger_thread, args=(q,))
lp.start()

对此:

logging.config.dictConfig(d)
lp = threading.Thread(target=logger_thread, args=(q,))
lp.start()
workers = []
for i in range(5):
    wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
    workers.append(wp)
    wp.start()

换出logging.FileHandler换成logging.handlers.RotatingFileHandler(用很小的maxBytes进行测试)，然后我遇到了这个错误.

我正在使用Windows和python 2.7. QueueHandler并非stdlib直到python 3.2的一部分，但我已从要点复制了源代码，它说是安全的.

我不明白为什么首先启动侦听器会有什么不同，也我不明白为什么除main之外的任何进程都试图访问文件.

解决方案

在子进程之前，切勿启动任何线程.当Python分叉时，线程和IPC状态将不会总是正确复制.

关于此有几种资源，仅Google提供了fork和threads.有些人声称他们可以做到，但是我不清楚它是否可以正常工作.

只需首先启动所有进程即可.

附加信息示例:

在Python中混合多处理和线程的状态

https://stackoverflow.com/a/6079669/4279

在您的情况下，可能是复制的打开文件句柄是问题所在，但是您仍然应该在线程之前(以及在打开任何以后要销毁的文件之前)启动子进程.

一些经验法则，从评论中总结出来:

• 子进程必须始终在同一进程创建的任何线程之前启动.

• multiprocessing.Pool会创建两个子进程和线程，因此一个人不能在第一个之后创建其他进程或池.

• 在创建进程或池时，文件不应已经打开. (在某些情况下可以，但是，例如，稍后再删除文件.)

• 子流程可以创建自己的线程和流程，并具有与上述相同的规则.

• 首先启动所有进程是最简单的方法

I'm converting a program to multiprocessing and need to be able to log to a single rotating log from the main process as well as subprocesses. I'm trying to use the 2nd example in the python cookbook Logging to a single file from multiple processes, which starts a logger_thread running as part of the main process, picking up log messages off a queue that the subprocesses add to. The example works well as is, and also works if I switch to a RotatingFileHandler.

However if I change it to start logger_thread before the subprocesses (so that I can log from the main process as well), then as soon as the log rotates, all subsequent logging generates a traceback with WindowsError: [Error 32] The process cannot access the file because it is being used by another process.

In other words I change this code from the 2nd example

workers = []
for i in range(5):
    wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
    workers.append(wp)
    wp.start()
logging.config.dictConfig(d)
lp = threading.Thread(target=logger_thread, args=(q,))
lp.start()

to this:

logging.config.dictConfig(d)
lp = threading.Thread(target=logger_thread, args=(q,))
lp.start()
workers = []
for i in range(5):
    wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
    workers.append(wp)
    wp.start()

and swap out logging.FileHandler for logging.handlers.RotatingFileHandler (with a very small maxBytes for testing) and then I hit this error.

I'm using Windows and python 2.7. QueueHandler is not part of stdlib til python 3.2 but I've copied the source code from Gist, which it says is safe to do.

I don't understand why starting the listener first would make any difference, nor do I understand why any process other than main would be attempting to access the file.

解决方案

You should never start any threads before subprocesses. When Python forks, the threads and IPC state will not always be copied properly.

There are several resources on this, just google for fork and threads. Some people claim they can do it, but it's not clear to me that it can ever work properly.

Just start all your processes first.

Example additional information:

Status of mixing multiprocessing and threading in Python

https://stackoverflow.com/a/6079669/4279

In your case, it might be that the copied open file handle is the problem, but you still should start your subprocesses before your threads (and before you open any files that you will later want to destroy).

Some rules of thumb, summarized by fantabolous from the comments:

• Subprocesses must always be started before any threads created by the same process.

• multiprocessing.Pool creates both subprocesses AND threads, so one mustn't create additional Processes or Pools after the first one.

• Files should not already be open at the time a Process or Pool is created. (This is OK in some cases, but not, e.g. if a file will be deleted later.)

• Subprocesses can create their own threads and processes, with the same rules above applying.

• Starting all processes first is the easiest way to do this

这篇关于python多处理日志记录:带有RotatingFileHandler“正在由另一个进程使用的文件"的QueueHandler；错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！