如何在SpringBatch中提高FlatFileItemReader的性能? [英] How to increase the performance of FlatFileItemReader in SpringBatch?

问题描述

我正在编写从平面文件读取，少量处理并将摘要写入输出文件的spring批处理.与阅读器相比，我的处理器和书写器相对较快.我正在使用FlatFileItemReader，并尝试了从50-1000开始的广泛提交间隔.我的批处理作业必须以更快的速度处理1000万条记录.请让我知道提高FlatFileItemReader速度的方法.粘贴到我的配置文件和Mapper类下面，读取该字段集并将其值设置为POJO bean.在此先多谢.

I am writing spring batch which reads from flat file, do little processing and write the summary to the output file. My processor and writer are relatively quicker compared to reader. I am using FlatFileItemReader and tried with wide range of commit intervals starting from 50-1000. My batch job has to process 10 millions records on a faster rate. Kindly let me know the ways to improve the speed rate of FlatFileItemReader. pasting below my config file and my Mapper class read the fieldset and set the values to POJO bean. Thanks a lot in advance.

BatchFileConfig.xml

BatchFileConfig.xml

<!-- Flat File Item Reader and its dependency configuration starts here -->
<bean id="flatFileReader" class="org.springframework.batch.item.file.FlatFileItemReader">
    <property name="resource" value="classpath:flatfiles/input_10KFile.txt" />
    <property name="encoding" value="UTF-8" />
    <property name="linesToSkip" value="1" />
    <property name="lineMapper">
        <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
            <property name="lineTokenizer">
                <bean
                    class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                    <property name="names"
                        value="var1,var2,var3,var4,var5,var6" />
                    <property name="delimiter" value="&#009;" />
                    <property name="strict" value="false" />
                </bean>
            </property>
            <property name="fieldSetMapper" ref="companyMapper">
            </property>
        </bean>
    </property>
</bean>

CompanyMapper.java

CompanyMapper.java

 public Company mapFieldSet(FieldSet fieldSet) throws BindException {
    logger.warn("Start time is "+System.currentTimeMillis());
    if (fieldSet != null) {
    Company company = new Company();
    company.setvar1(fieldSet.readString("var1"));
    company.setvar2(fieldSet.readInt("var2"));
    company.setvar3(fieldSet.readString("var3"));
    company.setvar4(fieldSet.readInt("var4"));
    company.setvar5(fieldSet.readInt("var5"));
    company.setvar6(fieldSet.readInt("var6"));
    return company;
    }
    return null;
}

推荐答案

我认为您无法大量加快处理速度:/ CompanyMapper已经是一个自定义实现，因此您可以考虑:

I think you can't speed-up process a lot :/ CompanyMapper is already a custom implementation so you can think:

1. 编写自定义的LineTokinizer + FieldSet对，以避免大量(有用的)检查和错误处理
2. 编写自定义BufferedReaderFactory来创建您自己的BufferedReader实现，该实现包装了一个自定义(且速度更快)的InputStream实现(为此请寻找Google)

1. write a custom LineTokinizer + FieldSet couple to avoid a lot of (useful) check and error handling
2. write a custom BufferedReaderFactory to create your own implementation of BufferedReader that wraps a custom (and faster) InputStream implementation (look Google for that)

这篇关于如何在SpringBatch中提高FlatFileItemReader的性能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！