我应该如何在Python中将HTTPHandler与RotatingFileHandler链接起来? [英] How should I chain HTTPHandler with RotatingFileHandler in Python?

问题描述

我需要创建一个系统，在该系统中，嵌入式系统中生成的日志消息将远程记录在服务器上，并存储在轮换的日志文件中.由于网络通信的限制，必须通过HTTP协议传输日志消息.该服务器已运行基于 Flask 的HTTP服务器，因此我想将日志记录系统与Flask集成基于网络的应用程序.

I needed to create a system, where log messages generated in an embedded system are logged remotely on a server and stored in the rotated log files. Due to restrictions in network communication, the log messages had to be transported via HTTP protocol. The server runs already a Flask based HTTP server, and therefore I wanted to integrate the logging system with Flask-based web applicattion.

Python logging 模块提供了专用的 HTTPhandler 类.基于logging文档，我创建了第一个实现.

The Python logging module offers the dedicated HTTPhandler class for that purpose. Basing on the logging documentation, I have created the first implementation.

服务器部分:

from flask import Flask
from flask import Response
from flask import request
import logging
import logging.handlers
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'logging server'

@app.route('/log', methods=['POST'])
def handle_log():
    if request.method == 'POST':
       rd=request.form.to_dict()
       record = logging.makeLogRecord(rd)
       log1.handle(record)
       return "OK"

log1=logging.getLogger('MTEST')
log1.setLevel(logging.DEBUG)
fh=logging.handlers.RotatingFileHandler('logs','a',maxBytes=10000000,backupCount=10)
formatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
rfh.setFormatter(formatter)
log1.addHandler(rfh)
log1.error("First error generated locally")
app.run()

客户端部分:

import logging, logging.handlers
myLogger = logging.getLogger('MTEST')
myLogger.setLevel(logging.DEBUG)
httpHandler = logging.handlers.HTTPHandler('localhost:5000',url='/log',method="POST")
myLogger.addHandler(httpHandler)

myLogger.info('Small info message')
myLogger.debug('Small debug message')
myLogger.erro('Small error message')

但是，在该实现中，服务器报告了以下错误(我复制了四个几乎相同的错误消息之一)，并且仅记录了本地生成的错误消息.

However, with that implementation the server reported the following errors (I copied one of four almost identical error messages) and only the locally generated error message was logged.

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 76, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python2.7/logging/handlers.py", line 156, in shouldRollover
    msg = "%s\n" % self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 741, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

我添加了打印传输到服务器的表单数据和创建的LogRecord对象的属性的信息.看来args属性是作为空的tupple传输的，而不是创建的. 为了解决这个问题，我修改了handle_log例程:

I have added printing of the form data transmitted to the server and the attributes of the created LogRecord object. It appeared, that the args attribute is transmitted as empty tupple and not created. To cure that I have modified the handle_log routine:

@app.route('/log', methods=['POST'])
def handle_log():
    if request.method == 'POST':
       rd=request.form.to_dict()
       rd['args']=""
       record = logging.makeLogRecord(rd)
       log1.handle(record)
       return "OK"

日志仍然无法正常运行，但是服务器产生的错误已更改:

Logging still wasn't working, but the errors produced by the server have changed:

Logged from file test1.py, line 9
127.0.0.1 - - [10/Jul/2018 23:52:51] "POST /log HTTP/1.0" 200 -
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 76, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python2.7/logging/handlers.py", line 156, in shouldRollover
    msg = "%s\n" % self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 741, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 467, in format
    record.asctime = self.formatTime(record, self.datefmt)
  File "/usr/lib/python2.7/logging/__init__.py", line 423, in formatTime
    ct = self.converter(record.created)
TypeError: a float is required

要检查什么地方出了问题，我添加了打印创建的LogRecord对象的过滤器:

To check, what is wrong, I have added the filter printing the created LogRecord object:

from flask import Flask
from flask import Response
from flask import request
import logging
import logging.handlers
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'logging server'

@app.route('/log', methods=['POST'])
def handle_log():
    if request.method == 'POST':
       rd=request.form.to_dict()
       rd['args']=""
       record = logging.makeLogRecord(rd)
       log1.handle(record)
       return "OK"

class myflt(logging.Filter):
  def filter(self,rec):
    print(rec.__dict__)
    return 1

log1=logging.getLogger('MTEST')
log1.setLevel(logging.DEBUG)
rfh=logging.handlers.RotatingFileHandler('logs','a',maxBytes=10000000,backupCount=10)
formatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
rfh.setFormatter(formatter)
log1.addHandler(rfh)
log1.addFilter(myflt())
log1.error("First error generated locally")
app.run()

之后，我可以比较为本地生成的消息创建的LogRecord对象:

After that, I could compare the LogRecord objects created for the locally generated message:

{'threadName': 'MainThread', 'name': 'MTEST', 'thread': 139678016341824, 'created': 1531259894.112536, 'process': 5595, 'processName': 'MainProcess', 'args': (), 'module': 'srv1', 'filename': 'srv1.py', 'levelno': 40, 'exc_text': None, 'pathname': 'srv1.py', 'lineno': 37, 'msg': 'First error generated locally', 'exc_info': None, 'funcName': '<module>', 'relativeCreated': 44.27504539489746, 'levelname': 'ERROR', 'msecs': 112.53595352172852}

对于远程生成的消息:

{'relativeCreated': u'12.3879909515', 'process': u'5597', 'module': u'test1', 'funcName': u'<module>', 'filename': u'test1.py', 'levelno': u'10', 'processName': u'MainProcess', 'lineno': u'9', 'msg': u'Small debug message', 'args': '', 'exc_text': u'None', 'name': u'MTEST', 'thread': u'140221336438592', 'created': u'1531259898.51', 'threadName': u'MainThread', 'msecs': u'511.312007904', 'pathname': u'test1.py', 'exc_info': u'None', 'levelname': u'DEBUG'}

看来，通过HTTPHandler传输日志记录会将所有数字转换为字符串，因此makeLogRecord函数不能正确地在服务器中重新创建LogRecord对象.

It appears, that the transmission of the log record via HTTPHandler converts all numbers into strings, and therefore the makeLogRecord function is not able to correctly recreate the LogRecord object in the server.

通过HTTPHandler传输日志消息的正确方法是什么，以便可以由例如远程计算机中的RotatingFileHandler正确处理?

What is the proper way to transmit the log messages via HTTPHandler so that they may be properly handled by e.g., RotatingFileHandler in a remote machine?

推荐答案

我发现了一些可行的解决方法. 在客户端中创建的带有LogRecord对象的属性的字典在传递到服务器之前会转换为JSON.然后，服务器将其解码并重新创建原始的LogRecord.

I have found something, that seems to be a viable workaround. The dictionary with atributes of the LogRecord object created in the client is converted to JSON before passing to the server. The server then decodes it and recreates the original LogRecord.

服务器部分:

from flask import Flask
from flask import Response
from flask import request
import logging
import logging.handlers
import json
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'logging server'

@app.route('/log', methods=['POST'])
def handle_log():
    if request.method == 'POST':
       rd=request.form.to_dict()
       rec=json.loads(rd['record'])
       record = logging.makeLogRecord(rec)
       log1.handle(record)
       return "OK"

class myflt(logging.Filter):
  def filter(self,rec):
    print(rec.__dict__)
    return 1

mfl=myflt()

log1=logging.getLogger('MTEST')
log1.setLevel(logging.DEBUG)
rfh=logging.handlers.RotatingFileHandler('logs','a',maxBytes=10000000,backupCount=10)
formatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
rfh.setFormatter(formatter)
log1.addHandler(rfh)
log1.addFilter(mfl)
log1.error("First error generated locally")
app.run()

客户端部分:

import logging, logging.handlers
import json

class myHTTPHandler(logging.handlers.HTTPHandler):
  def mapLogRecord(self,record):
    trec={'record':json.dumps(record.__dict__)}
    return trec

myLogger = logging.getLogger('MTEST')
myLogger.setLevel(logging.DEBUG)
httpHandler = myHTTPHandler('localhost:5000',url='/log',method="POST")
myLogger.addHandler(httpHandler)

myLogger.info('Small info message')
myLogger.debug('Small debug message')
myLogger.error('Small error message')

上面的实现正常工作，但是似乎不必要地复杂. 有没有更简单的方法可以使用HTTPHandler通过HTTP协议与远程RotatingFileHandler进行通信?

The above implementation works correctly, however it seems to be unnecessarily complicated. Is there any simpler way to use HTTPHandler to communicate with the remote RotatingFileHandler via HTTP protocol?

这篇关于我应该如何在Python中将HTTPHandler与RotatingFileHandler链接起来?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！