如何在同一配置文件中混合日志记录处理程序(文件和定时)和压缩日志? [英] How to mix logging Handlers (File & Timed) and compress log in the same config file?
问题描述
我需要在json中准备一个日志记录配置文件，该文件会按时间，大小旋转，并为我的应用程序中所有模块旋转的压缩文件进行旋转(我现在卡住了).我想使用一个sigle json配置文件来执行此操作，这是我当前的文件，但是此配置仅按时间旋转:
I need to prepare a logging config file in json which rotate it by time, size and compress de file rotated for all modules in my app (I'm stuck now). I want to do it using a sigle json config file, this is my current file, however this configuration only rotate by time:
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"simple": {
"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S"
}
},
"handlers": {
"my_rotate": {
"level": "DEBUG",
"class": "logging.handlers.TimedRotatingFileHandler",
"formatter": "simple",
"when": "D",
"interval": 1,
"backupCount": 5,
"filename": "/var/log/test.log",
"encoding": "utf8"
}
},
"loggers": {
"my_module": {
"level": "DEBUG",
"handlers": ["my_rotate"]
}
},
"root": {
"level": "DEBUG",
"handlers": ["my_rotate"],
"propagate": false
}
}
如果我在处理程序部分(logging.handlers.RotateFileHandler)中添加其他处理程序，以按大小轮换结果日志文件中的重复消息.
If I add other handler in handlers section (logging.handlers.RotateFileHandler) to rotate by size the result log file duplicate al messages.
我的主程序使用以下方法获取记录器属性:
My main program get logger properties using:
config_json_file = open(os.path.join(os.path.dirname(os.path.realpath(__file__)), "logging.json"))
logging_data = json.load(config_json_file)
logging.config.dictConfig(logging_data)
我的模块像这样调用记录器:
My modules call the logger like:
class StoreData(object):
def __init__(self):
self.logger = logging.getLogger(__name__)
如何在此文件中混合我的三个要求(按大小和时间旋转，并压缩旋转的文件)?
How could I mix my three requirements (rotate by size and time, and compress rotated file) in this file?
很多东西！
更新1
按照建议@Bakuriu，我创建了一个新类来覆盖所需的方法，以便能够压缩旋转的文件:
As suggested @Bakuriu I've created a new class to override needed methods to be able to compress rotated files:
class MyRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
""" My rotating file hander to compress rotated file """
def __init__(self, **kwargs):
logging.handlers.TimedRotatingFileHandler.__init__(self, **kwargs)
def rotate(self, source, dest):
""" Compress rotated log file """
os.rename(source, dest)
f_in = open(dest, 'rb')
f_out = gzip.open("%s.gz" % dest, 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
os.remove(dest)
这样新"类称为fron记录配置文件，在其中我添加了新的处理程序:
So that "new" class is called fron logging config file where I added a new handler:
"handlers": {
"my_rotate": {
"level": "DEBUG",
"class": "myMainPackage.MyRotatingFileHandler",
"formatter": "simple",
"when": "D",
"interval": 1,
"backupCount": 5,
"filename": "/var/log/test.log",
"encoding": "utf8"
}
在下一次更新中，我将添加一个方法来覆盖所需的方法，并能够将时间和文件大小的旋转混合在一起.
In the next update I'll add a method to override the needed method(s), and be able to mix time and file size rotation together.
更新2
好吧，我已经覆盖了"shouldRollover"方法，因此我将时间和文件大小混合使用了.
Well, I've overrided "shouldRollover" method, thus I've mix the time and file size for rotation.
class MyRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
""" My rotating file hander to compress rotated file """
def __init__(self, filename, mode='a', maxBytes=0, backupCount=0, encoding=None,
delay=0, when='h', interval=1, utc=False):
if maxBytes > 0:
mode = 'a'
logging.handlers.TimedRotatingFileHandler.__init__(
self, filename, when, interval, backupCount, encoding, delay, utc)
self.maxBytes = maxBytes
self.backupCount = backupCount
def shouldRollover(self, record):
""" Determine if rollover should occur. """
# Check rollover by size
if self.stream is None: # delay was set...
self.stream = self._open()
if self.maxBytes > 0: # are we rolling over?
msg = "%s\n" % self.format(record)
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
if self.stream.tell() + len(msg) >= self.maxBytes:
return 1
# Check rollover by time
t = int(time.time())
if t >= self.rolloverAt:
return 1
return 0
def rotate(self, source, dest):
""" Compress rotated log file """
os.rename(source, dest)
f_in = open(dest, 'rb')
f_out = gzip.open("%s.gz" % dest, 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
os.remove(dest)
然后在json中修改我的日志记录配置文件以按天轮换时间，但要以秒为单位...这意味着 86400s (一天中的秒数)，大小限制为 5Mb :
And modify my logging config file in json to rotate time by day but check in seconds... it means 86400s (seconds in a day) and a size limit of 5Mb:
"handlers": {
"my_rotate_timed": {
"level": "DEBUG",
"class": "myMainPackage.MyRotatingFileHandler",
"formatter": "simple",
"when": "S",
"interval": 86400,
"backupCount": 5,
"maxBytes": 5242880,
"filename": "/var/log/test.log",
"encoding": "utf8"
}
这种方法重用了TimedRotationFileHandler的其他方法，并且如果 maxBytes 调用了旋转，那么它将使用从 init TimedRotationFileHandler继承的后缀方法，其格式为:％Y-％m-％d_％H-％M-％S".这就是我使用"{when =" S，interval = 86400}"而不是"{when =" D，interval = 1}""
This way reuses the others methods from TimedRotationFileHandler and if rotation is called by maxBytes, then it uses suffix in seconds inherited from init TimedRotationFileHandler method, which format is: "%Y-%m-%d_%H-%M-%S" . This is the reason I've used "{when="S", interval=86400}" and not "{when="D", interval=1}"
推荐答案
这是预料之中的.如果您添加处理程序，则该处理程序要么不会生成消息(由于过滤)，所以什么都不会改变，或者会写入这些消息(因此将它们复制).
This is to be expected. If you add a handler this handler either wont produce messages (due to filtering) and so nothing will change or it will write those messages (thus duplicating them).
TimedRotatingFileHandler和RotatingFileHandler仅分别支持按时间和大小旋转.并非两者都在同一时间.
The TimedRotatingFileHandler and the RotatingFileHandler only support, respectively, rotating by time and by size. Not both at the same time.
AFAIK 没有内置的方法来实现所需的目标，因此仅使用配置文件就无法实现所需的目标，而必须编写一些代码来组合功能.
AFAIK there is no built-in way to achieve what you want, so using only the configuration file you wont be able to achieve what you want, you have to code something to put together the functionality.
考虑阅读如何在记录食谱.如果将此处理程序类保存在文件mypackage.myrotatinghandler中，则可以指定:
Consider reading how to create a new rotating handler in the Logging Cookbook. If you save this handler class in a file mypackage.myrotatinghandler you can then specify:
class: 'mypackage.myrotatinghandler.MyHandler'
在配置文件中.
请注意，要在保存文件时添加压缩，只需使用保存压缩文件的功能来设置旋转处理程序的rotator属性即可.来自上面的链接:
Note that, to add compression when saving a file it is sufficient to set the rotator attribute of the rotating handler with a function that saves a compressed file. Taken from the link above:
def namer(name):
return name + ".gz"
def rotator(source, dest):
with open(source, "rb") as sf:
data = sf.read()
compressed = zlib.compress(data, 9)
with open(dest, "wb") as df:
df.write(compressed)
os.remove(source)
rh = logging.handlers.RotatingFileHandler(...)
rh.rotator = rotator
rh.namer = namer
rh处理程序将像普通的RotatingFileHandler一样工作，但还会压缩日志.
The rh handler will behave as a normal RotatingFileHandler but also compress the logs.
但是，设置条件进行轮换需要您重写处理程序的各个部分.您可以阅读 logging.handlers 模块的源代码了解内置处理程序的实现方式.
However setting the conditions to do a rotation require that you re-write parts of the handler. You may read the sources for the logging.handlers module to see how the built-in handlers are implemented.
这篇关于如何在同一配置文件中混合日志记录处理程序(文件和定时)和压缩日志?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
