Spring Batch - 读取多行日志消息 [英] Spring Batch - Reading multiple line log message

问题描述

在我们配置了 spring 集成的 spring 批处理应用程序中，我面临将多行日志消息作为单个消息读取的问题，该应用程序必须将多行日志消息(示例异常堆栈跟踪)作为单个消息读取，稍后它必须处理和分类消息以进一步索引.每行由其时间戳标识(上面提到的模式，即 DATE_PATTERN)并且它可能会继续多行，我正在尝试继续读取消息，直到我通过覆盖看到另一个时间戳当第二行到达 preProcess 方法时，来自 SimpleRecordSeparatorPolicy 的 isEndOfRecord 方法我为 isEndOfRecord 返回 true 但这没有按预期工作，有人可以帮助我通过识别时间戳模式来读取提到的日志文件吗?

I am facing a problem to read multi-line log message as a single message in our spring batch application configured with spring integration, this application has to read multiline log message (example exception stack trace) as a single message, later it has to process and classify the message for further indexing. Each line is identified by its timestamp (pattern mentioned above i.e. DATE_PATTERN) and it may continue mutltiple lines, I am trying to continue reading a message until I see another timestamp by overriding isEndOfRecord method from SimpleRecordSeparatorPolicy when second line reaches in preProcess method I am returning true for isEndOfRecord but this is not working as expected, could any one help me to read the mentioned log file by identifying the timestamp pattern?

我使用 org.springframework.batch.item.file.FlatFileItemReader 和 org.springframework.batch.item.file.mapping.PassThroughLineMapper 作为映射器.

I am using org.springframework.batch.item.file.FlatFileItemReader, and org.springframework.batch.item.file.mapping.PassThroughLineMapper as mapper.

请查看完整消息，

1) 日志消息文件:sample-message-test.log

1) log message file :sample-message-test.log

2013-10-19 07:05:32.253 [My First Class..] LOG LEVEl  first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 
first-message-line-2 first-message-line-2 first-message-line-2 
first-message-line-3 first-message-line-3 first-message-line-3 
first-message-line-4 first-message-line-4 first-message-line-4 
first-message-line-5 first-message-line-5 
first-message-line-6 
2013-10-19 07:05:32.257 [My Second Class..] LOG LEVEl  second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 
second-message-line-2 second-message-line-2 second-message-line-2 
second-message-line-3 second-message-line-3 second-message-line-3 
second-message-line-4 second-message-line-4 second-message-line-4 
second-message-line-5 second-message-line-5 
second-message-line-6
2013-10-19 07:05:32.259 [My Third Class..] LOG LEVEl  third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 
third-message-line-2 third-message-line-2 third-message-line-2 
third-message-line-3 third-message-line-3 third-message-line-3 
third-message-line-4 third-message-line-4 third-message-line-4 
third-message-line-5 third-message-line-5 
third-message-line-6

2) 批量配置文件

<batch:job id="fileReadingJob">
        <batch:step id="flatFileReadingStep">
            <batch:tasklet >
                <batch:chunk reader="reader" writer="writer" commit-interval="10" />
            </batch:tasklet>
        </batch:step>
    </batch:job>

    <bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"  scope="step">
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper"/>
        </property>
        <property name="bufferedReaderFactory">
            <bean class="org.springframework.batch.item.file.DefaultBufferedReaderFactory"/>
        </property>
        <property name="recordSeparatorPolicy" >
            <bean class="com.batchlog.explorer.batchio.FlatFileRecordSeperationPolicy"/>
        </property>
        <property name="resource" value="file:///#{systemProperties['logfolder']}/#{jobParameters['inputfile']}" />
    </bean>
    <bean id="writer" class="com.batchlog.explorer.batchio.FlatFileWriter" scope="step"/>
........

3)

public class FlatFileRecordSeperationPolicy extends SimpleRecordSeparatorPolicy {

    public static final String STARTING_OF_THE_LINE = "-STARTING_OF_THE_LINE-";
    public static final String CONTINUATION_OF_THE_FILE  = "-CONTINUATION_OF_THE_FILE-";
    public static final String END_OF_THE_LINE = "-END_OF_THE_LINE-";
    public static final String END_OF_THE_LINE_CHARACER = " \n ";
    public static final String DATE_PATTERN ="^(?>\\d\\d){1,2}-(?:0?[1-9]|1[0-2])-(\\s)?(?:2[0123]|[01][0-9]):? (?:[0-5][0-9])(?::?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01][0-9])(?::?(?:[0-5][0-9])))?.*?";


    @Override
        public boolean isEndOfRecord(String line) {
            if(line.matches(DATE_PATTERN) || line.startsWith(STARTING_OF_THE_LINE)
                            || line.contains(CONTINUATION_OF_THE_FILE) || line.startsWith(END_OF_THE_LINE)){
                if(isNextLineStarts(line) || line.startsWith(END_OF_THE_LINE)){
                    return true;//to break line
                }
        }
        return false; //to conitnue line

    private boolean isNextLineStarts(String preProcessOfLine){
            if(preProcessOfLine.contains(CONTINUATION_OF_THE_FILE) && !preProcessOfLine.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = preProcessOfLine.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    return true;
                }
            }
            return false;
    }
    @Override
        public String preProcess(String line) {
            if(line.matches(DATE_PATTERN) && !line.contains(CONTINUATION_OF_THE_FILE)){
                line = new StringBuilder(STARTING_OF_THE_LINE).append(line).toString();
            }else if(line.startsWith(STARTING_OF_THE_LINE) && !line.contains(CONTINUATION_OF_THE_FILE)){
                line =  new StringBuilder(line.substring(STARTING_OF_THE_LINE.length())).append(CONTINUATION_OF_THE_FILE).toString();
            }else if(line.contains(CONTINUATION_OF_THE_FILE) && !line.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = line.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    line = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();//.append(lines[1]).toString();
                }else{
                    line = new StringBuilder(lines[0]).append(lines[1]).append(CONTINUATION_OF_THE_FILE).toString();
                }
            }
                return super.preProcess(line);
    }
    @Override
        public String postProcess(String record) {
            if(record.startsWith(END_OF_THE_LINE)){
                record = new StringBuilder(record.substring(END_OF_THE_LINE.length())).toString();
            }else if(record.contains(CONTINUATION_OF_THE_FILE) && !record.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = record.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    record = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();
                }else{
                    record = new StringBuilder(lines[0]).append(lines[1]).toString();
                }
            }
            return super.postProcess(record);
    }

推荐答案

按照 multiorder-line 示例 或如 这篇文章.

Write your own ItemReader as described in multiorder-line example or as described in this post.

这篇关于Spring Batch - 读取多行日志消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！