Python多重处理-logging.FileHandler对象引发PicklingError [英] Python multiprocessing - logging.FileHandler object raises PicklingError

问题描述

logging模块和multiprocessing作业的处理程序似乎不混合:

It seems that handlers from the logging module and multiprocessing jobs do not mix:

import functools
import logging
import multiprocessing as mp

logger = logging.getLogger( 'myLogger' )
handler = logging.FileHandler( 'logFile' )

def worker( x, handler ) :
    print x ** 2

pWorker = functools.partial( worker, handler=handler )

#
if __name__ == '__main__' :
    pool = mp.Pool( processes=1 )
    pool.map( pWorker, range(3) )
    pool.close()
    pool.join()

出局:

cPickle.PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

如果我将pWorker替换为以下方法之一，则不会引发错误

If I replace pWorker be either one of the following methods, no error is raised

# this works
def pWorker( x ) :
    worker( x, handler )

# this works too
pWorker = functools.partial( worker, handler=open( 'logFile' ) )

我不太了解PicklingError.是否是因为类logging.FileHandler的对象不可选? (我用谷歌搜索，但没有找到任何东西)

I don't really understand the PicklingError. Is it because objects of class logging.FileHandler are not pickable? (I googled it but didn't find anything)

推荐答案

FileHandler对象内部使用threading.Lock来同步线程之间的写入.但是，threading.Lock返回的thread.lock对象不能被腌制，这意味着它不能在进程之间发送，而必须通过pool.map发送给子对象.

The FileHandler object internally uses a threading.Lock to synchronize writes between threads. However, the thread.lock object returned by threading.Lock can't be pickled, which means it can't be sent between processes, which is required to send it to the child via pool.map.

multiprocessing文档中有一节讨论了使用multiprocessing进行记录的方式.

There is a section in the multiprocessing docs that talks about how logging with multiprocessing works here. Basically, you need to let the child process inherit the parent process' logger, rather than trying to explicitly pass loggers or handlers via calls to map.

不过请注意，在Linux上，您可以执行以下操作:

Note, though, that on Linux, you can do this:

from multiprocessing import Pool
import logging

logger = logging.getLogger( 'myLogger' )


def worker(x):
    print handler
    print x **2 

def initializer(handle):
    global handler
    handler = handle

if __name__ == "__main__":
    handler = logging.FileHandler( 'logFile' )
    #pWorker = functools.partial( worker, handler=handler )
    pool = Pool(processes=4, initializer=initializer, initargs=(handler,))
    pool.map(worker, range(3))
    pool.close()
    pool.join

initializer/initargs用于在池的每个子进程启动后立即运行一次方法.在Linux上，由于os.fork的工作方式，它允许处理程序通过继承进入子级.但是，在Windows上将无法使用；因为它不支持os.fork，所以仍然需要腌制handler才能通过initargs传递它.

initializer/initargs are used to run a method once in each of the pool's child processes as soon as they start. On Linux this allows the handler to be into the child via inheritance, thanks to the way os.fork works. However, this won't work on Windows; because it lacks support for os.fork, it would still need to pickle handler to pass it via initargs.

这篇关于Python多重处理-logging.FileHandler对象引发PicklingError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！