如何在Flume中处理多行日志条目 [英] How to handle multiline log entries in Flume
问题描述
我刚开始玩Flume.我有一个关于如何将多行日志条目作为一个事件处理的问题.类似于错误情况下的堆栈跟踪. 例如，将以下内容视为一个事件，而不是每行一个事件
I have just started playing with Flume. I have a question on how to handle log entries that are multiline, as a single event. Like stack traces during error conditions. For example, treat the below as a single event rather than one event for each line
2013-04-05 05:00:41,280错误(ClientRequestPool-PooledExecutionEngine-Id#4)[com.ms.fw.rexs.gwy.api.service.AbstractAutosysJob]作业失败，导致228794 java.lang.NullPointerException 在com.ms.fw.rexs.core.impl.service.job.ReviewNotificationJobService.createReviewNotificationMessageParameters(ReviewNotificationJobService.java:138)处 ....
2013-04-05 05:00:41,280 ERROR (ClientRequestPool-PooledExecutionEngine-Id#4 ) [com.ms.fw.rexs.gwy.api.service.AbstractAutosysJob] job failed for 228794 java.lang.NullPointerException at com.ms.fw.rexs.core.impl.service.job.ReviewNotificationJobService.createReviewNotificationMessageParameters(ReviewNotificationJobService.java:138) ....
我已将源配置为spooldir类型.
I have configured the source to a spooldir type.
谢谢 苏曼
推荐答案
您需要研究扩展假脱机源使用的行反序列化器，一种简单(但可能有缺陷)的方法是在换行符上划界，但合并行在前一行前加上一定数量的空格作为前缀.
You'll want to look into extending the line deserializer used by spool source, one simple (but potentially flawed) approach would be delimit on newlines, but combine lines that are prefixed with a set number of spaces to the previous line.
事实上，已经有一个带有补丁的Jira问题:
In fact there is already a Jira issue for this with a patch:
这篇关于如何在Flume中处理多行日志条目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
