Windows中具有多处理功能的Python日志记录 [英] Python Logging with Multiprocessing in Windows
问题描述
我有一个相当大的Python项目,该项目当前在Linux上运行,但我正在尝试扩展到Windows.我将代码简化为一个完整的示例,可以运行该示例来说明我的问题:我有两个类,Parent和Child.首先初始化父级,创建一个记录器,并生成一个子级进行工作:
I have a rather large Python project which currently runs on Linux but I am trying to expand to Windows. I've reduced the code to a full example which can be run to illustrate my problems: I have two classes, Parent and Child. Parent is initialized first, creates a logger, and spawns a Child to do work:
import logging
import logging.config
import multiprocessing
class Parent( object ):
def __init__(self, logconfig):
logging.config.dictConfig(logconfig)
self.logger = logging.getLogger(__name__)
def spawnChild(self):
self.logger.info('One')
c = Child(self.logger)
c.start()
class Child(multiprocessing.Process):
def __init__(self, logger):
multiprocessing.Process.__init__(self)
self.logger = logger
def run(self):
self.logger.info('Two')
if __name__ == '__main__':
p = Parent({
'version':1,
"handlers": {
"console": {
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout"
},
},
"root": {
"level": "DEBUG",
"handlers": [
"console",
]
}
}
)
p.spawnChild()
在linux(特别是ubuntu 12.04)上,我得到以下(预期)输出:
On linux (specifically, ubuntu 12.04), I get the following (expected) output:
user@ubuntu:~$ python test.py
One
Two
但是,在Windows(特别是Windows 7)上,它会由于出现酸洗错误而失败:
But, on Windows (specifically, Windows 7), it fails with a pickling error:
C:\>python test.py
<snip>
pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock
问题归结于Windows缺少真正的fork,因此在线程之间发送对象时必须对对象进行腌制.但是,记录器不能被腌制.我尝试使用__getstate__和__setstate__避免酸洗,并在Child中按名称引用:
The problem comes down to Windows' lack of a true fork, so objects have to be pickled when sent between threads. But, the logger can't be pickled. I've tried using __getstate__ and __setstate__ to avoid pickling, and reference by name in Child:
def __getstate__(self):
d = self.__dict__.copy()
if 'logger' in d.keys():
d['logger'] = d['logger'].name
return d
def __setstate__(self, d):
if 'logger' in d.keys():
d['logger'] = logging.getLogger(d['logger'])
self.__dict__.update(d)
这与以前一样在Linux中有效,现在Windows不会因为PicklingError而失败.但是,我的输出仅来自父级:
This works in Linux just as before, and now Windows won't fail with the PicklingError. However, my output is only from Parent:
C:\>python test.py
One
C:\>
尽管没有消息抱怨找不到用于处理程序'__main__'的记录器"或任何其他错误消息,但孩子似乎无法使用记录器.我环顾四周,有一些方法可以完全重组我的程序登录方式,但这显然是不得已的方法.我希望我只是缺少一些明显的东西,而人群的智慧可以向我指出.
It seems that the child is unable to use the logger, despite no message complaining "No logger could be found for handler '__main__'" or any other error message. I've looked around and there are means by which I could completely restructure how I log in my program, but that's obviously a last resort. I'm hoping that I'm just missing something obvious, and that the wisdom of the crowd can point it out to me.
推荐答案
在大多数情况下,Logger
对象是不可拾取的,因为它们在内部使用不可拾取的theading.Lock
和/或file
对象.您尝试的解决方法确实避免了腌制logger
,但是最终在子进程中创建了一个完全不同的Logger
,而该子进程恰好与父进程中的Logger
具有相同的名称;您拨打的logging.config
呼叫的效果会丢失.要获得所需的行为,您需要在子进程中重新创建记录器,然后调用logging.config.dictConfig
:
In most cases, Logger
objects are not picklable, because they use unpicklable theading.Lock
and/or file
objects internally. Your attempted workaround does avoid pickling the logger
, but it ends up creating a completely different Logger
in the child process, which happens to have the same name as the Logger
in the parent; the effects of logging.config
call you made are lost. To get the behavior you want you'll need to need to recreate the logger in the child process and re-call logging.config.dictConfig
:
class Parent( object ):
def __init__(self, logconfig):
self.logconfig = logconfig
logging.config.dictConfig(logconfig)
self.logger = logging.getLogger(__name__)
def spawnChild(self):
self.logger.info('One')
c = Child(self.logconfig)
c.start()
class Child(multiprocessing.Process):
def __init__(self, logconfig):
multiprocessing.Process.__init__(self)
self.logconfig = logconfig
def run(self):
# Recreate the logger in the child
logging.config.dictConfig(self.logconfig)
self.logger = logging.getLogger(__name__)
self.logger.info('Two')
或者,如果您想继续使用__getstate__
/__setstate__
:
Or, if you want to keep using __getstate__
/__setstate__
:
class Parent( object ):
def __init__(self, logconfig):
logging.config.dictConfig(logconfig)
self.logger = logging.getLogger(__name__)
self.logconfig = logconfig
def spawnChild(self):
self.logger.info('One')
c = Child(self.logger, self.logconfig)
c.start()
class Child(multiprocessing.Process):
def __init__(self, logger, logconfig):
multiprocessing.Process.__init__(self)
self.logger = logger
self.logconfig = logconfig
def run(self):
self.logger.info('Two')
def __getstate__(self):
d = self.__dict__.copy()
if 'logger' in d:
d['logger'] = d['logger'].name
return d
def __setstate__(self, d):
if 'logger' in d:
logging.config.dictConfig(d['logconfig'])
d['logger'] = logging.getLogger(d['logger'])
self.__dict__.update(d)
这篇关于Windows中具有多处理功能的Python日志记录的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!