Python从日志文件解析 [英] Python parse from log file

问题描述

我的日志文件很大，容量很大.我如何只接收json字符串，仅当下一行出现错误但在上一行的'_____GP D_____之后出现错误时，才获取json字符串?

I have a big log file with huge volume. How do i take only the json string, just the json string only when there is an error in the next line but after '_____GP D_____' in the previous line?

2017-04-22T11:27:11+06:00 smth.com pgp: [16136]: INFO:modules.gp.helpers.parameter_getter:_____GP D_____
2017-04-22T11:27:11+06:00 smth.com pgp: [16136]: {'D': 't12', 'telephone': None, 'from_time': '2016-04-22 11:30', 'C': 'C12', 'to_time': '2016-04-22 11:40', 'email': None}
2017-04-22T11:27:11+06:00 smth.com pgp: [16136]: INFO:tornado.access:200 POST /gp/C (192.168.1.240) 15.77ms

2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: INFO:modules.security.authentication:LOADING USER...
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: INFO:modules.gp.helpers.parameter_getter:_____GP D_____
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: {'D': 'testim12', 'telephone': None, 'from_time': '2017-04-20 17:30', 'C': 'CnGP13', 'to_time': '2017-04-22 21:40', 'email': None}
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: ERROR:modules.common.actionexception:ActionError: [{'from': 'time is already passed'}]
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: Traceback (most recent call last):
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]:   File "/app/src/modules/base/actions/base_action.py", line 96, in do_action
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]:     self._produce_response()
2017-04-22T11:28:19+06:00 smth.com pgp: [16136]: modules.common.actionexception.ActionValidationErr: []

例如，我想要的该日志文件中的

for example from this log file i want

'{'D':'testim12'，'电话':无，'from_time':'2017-04-20 17:30'，'C':'CnGP13'，'to_time':'2017-04 -22 21:40'，'email':None}'.

'{'D': 'testim12', 'telephone': None, 'from_time': '2017-04-20 17:30', 'C': 'CnGP13', 'to_time': '2017-04-22 21:40', 'email': None}'.

只有当我有例外时，下一行才会出现'ERROR:modules.common.actionexception:ActionError:'吗?我该怎么办?

Only when i have an exception, ' ERROR:modules.common.actionexception:ActionError:' in the next line? how do i do it?

推荐答案

与昨天遇到的一个问题，只需在选择该行之前进行其他检查-例如，检查下一行是否包含]: ERROR:字符串:

That's the same problem as the one you had yesterday, just with an additional check before selecting the line - for example checking if the next line contains ]: ERROR: string:

found_line = None  # store for our matched line
with open("input.log", "r") as f:  # open your log file
    for line in f:  # read it line by line
        if line.rstrip()[-14:] == "_____GP D_____":  # if a line ends with our string...
            found_line = next(f).rstrip()  # grab the next line as our potential candidate
            if next(f).find("]: ERROR:") != -1:  # if the next line contains an error marker
                break  # match found, break out as we don't need to search any more...
            else:  # the next line wasn't an error...
                found_line = None  # ... reset the potential result and continue searching

但是，由于您的found_line实际上将包含整行(包括时间戳记)，因此您需要首先将其删除，而这一切都取决于记录器的设置方式.根据您的数据，一种合理的方法是跳过前39个字符(<date-time> smth.com pgp:)，并在下一个冒号之后提取所有内容，并假设以下括号中的数字可以更改(如果不能更改，则可以删除-前n个字符并用它完成):

However, since your found_line would actually contain the whole line (including the timestamp), you need to first strip that out, and that all depends on how your logger is set. A reasonable way, based on your data is to skip the first 39 characters (<date-time> smth.com pgp:) and pick up everything after the next colon, assuming that the number in the following brackets can change (if not - you can just strip out the first n characters and be done with it):

if found_line:
    found_line = found_line[found_line.find(":", 39) + 1:].strip()

请注意，如果某些记录的数据包含准确的模式，则错误"检查可能会失败-如果您要磨合，可以尝试使用类似的技术将JSON移出日志行，并检查它是否以ERROR:开头.

Beware, tho, that the 'error' check might fail if some of the logged data contains that exact pattern - if you want to hone in on it you can try using the similar technique that we use to lift the JSON out of the log line and check if it begins with ERROR:.

您还应该尝试自己做事，而不是盲目地从SO中复制代码-这样您将不会学到很多东西.

You should also try doing things on your own instead of blindly copying code from SO - you won't learn much this way.

这篇关于Python从日志文件解析的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！