用Python编写仅附加gzip压缩的日志文件 [英] Writing append only gzipped log files in Python
问题描述
我正在构建一个服务，在该服务中，我记录了来自多个来源的纯文本格式的日志(每个来源一个文件).我不打算旋转这些日志，因为它们必须永远存在.
I am building a service where I log plain text format logs from several sources (one file per source). I do not intend to rotate these logs as they must be around forever.
为使这些文件永久变小，我希望我能即时将其压缩.由于它们是日志数据，因此文件压缩得很好.
To make these forever around files smaller I hope I could gzip them in fly. As they are log data, the files compress very well.
在Python中写仅追加的gzip压缩文本文件的一种好方法是什么，以便以后在服务打开和关闭时可以恢复写操作?我并不担心丢失几行，但是如果gzip容器本身发生故障并且文件变得不可读，那不是不.
What is a good approach in Python to write append-only gzipped text files, so that the writing can be later resumed when service goes on and off? I am not that worried about losing few lines, but if gzip container itself breaks down and the file becomes unreadable that's no no.
此外，如果不行，如果不值得麻烦的话，我可以简单地将它们以纯文本格式编写而无需gzip.
Also, if it's no go, I can simply write them in as plain text without gzipping if it's not worth of the hassle.
推荐答案
注意:在Unix系统上，您应该认真考虑使用为该确切任务编写的外部程序:
Note: On unix systems you should seriously consider using an external program, written for this exact task:
-
logrotate(旋转，压缩并通过邮件发送系统日志)
您可以将轮换次数设置得如此之高，以使第一个文件在100年内被删除左右.
You can set the number of rotations so high, that the first file would be deleted in 100 years or so.
在Python 2中，logging.FileHandler采用关键字参数encoding，可以将其设置为bz2或zlib.
In Python 2, logging.FileHandler takes an keyword argument encoding that can be set to bz2 or zlib.
这是因为logging 使用 codecs模块，该模块又将bz2(或zlib)视为 encoding :
This is because logging uses the codecs module, which in turn treats bz2 (or zlib) as encoding:
>>> import codecs
>>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2") as fh:
... fh.write("Hello World\n")
$ bzcat on-the-fly-compressed.txt.bz2
Hello World
Python 3版本(尽管文档提及 bz2作为别名，实际上您必须使用bz2_codec-至少使用3.2.3):
Python 3 version (although the docs mention bz2 as alias, you'll actually have to use bz2_codec - at least w/ 3.2.3):
>>> import codecs
>>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2_codec") as fh:
... fh.write(b"Hello World\n")
$ bzcat on-the-fly-compressed.txt.bz2
Hello World
这篇关于用Python编写仅附加gzip压缩的日志文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
