如何使用FlatFileItemReader和chunks在CSV中跳过空行 [英] How to skip blank lines in CSV using FlatFileItemReader and chunks

问题描述

我正在使用FlatFileItemReader处理CSV文件。

有时我在输入文件中得到空行。

<当这发生时，整个步骤停止。我想跳过这些行，并继续正常。

我试图添加异常处理程序到捕获的execption，整个步骤弯曲：

  @Bean 
 public Step processSnidUploadedFileStep（）{
 return stepBuilderFactory.get processSnidFileStep）
。< MyDTO，MyDTO> chunk（numOfProcessingChunksPerFile）
 .reader（snidFileReader（OVERRIDDEN_BY_EXPRESSION））
 .processor（manualUploadAsyncItemProcessor（））
 .writer（manualUploadAsyncItemWriter （））
。listener（logProcessListener）
 .throttleLimit（20）
 .taskExecutor（infrastructureConfigurationConfig.taskJobExecutor（））
 .exceptionHandler（（context，throwable） - > logger .error（Skipping record on file。cause =+（（FlatFileParseException）throwable）.getCause（）））
 .build（） 
} 
  

由于我使用块并且异常被捕获发生的是整个块被跳过（该块可能包含CSV文件上的有效行，并且它们被跳过aswell）

感谢您，
ray。

编辑我的代码后。仍然不跳过：

  public Step processSnidUploadedFileStep（）{
 SimpleStepBuilder< MyDTO，MyDTO> builder = new SimpleStepBuilder< MyDTO，MyDTO>（stepBuilderFactory.get（processSnidFileStep））; 
 return builder 
。< PushItemDTO，PushItemDTO> chunk（numOfProcessingChunksPerFile）
 .faultTolerant（）skip（FlatFileParseException.class）
 .reader（snidFileReader（OVERRIDDEN_BY_EXPRESSION））
 
 
 .taskExecutor（infrastructureConfigurationConfig.taskJobExecutor（））
。 ）
 .build（）; 
} 
  

解决方案

$ c> SimpleRecordSeparatorPolicy 这是告诉读者跳过空白行。这样，我们读取100条记录，即3条是空白行，这些行被忽略，没有例外，它写入97条记录。

这里是代码：

  package com.my.package; 
 
 import org.springframework.batch.item.file.separator.SimpleRecordSeparatorPolicy; 
 
 public class BlankLineRecordSeparatorPolicy extends SimpleRecordSeparatorPolicy {
 
 @Override 
 public boolean isEndOfRecord（final String line）{
 return line.trim（）。length ）！= 0&& super.isEndOfRecord（line）; 
} 
 
 @Override 
 public String postProcess（final String record）{
 if（record == null || record.trim（）。length（） = 0）{
 return null; 
} 
 return super.postProcess（record）; 
} 
 
} 
  

p>

  package com.my.package; 
 
 import org.springframework.batch.core.configuration.annotation.StepScope; 
 import org.springframework.batch.item.file.FlatFileItemReader; 
 import org.springframework.batch.item.file.mapping.DefaultLineMapper; 
 import org.springframework.batch.item.file.transform.DelimitedLineTokenizer; 
 import org.springframework.stereotype.Component; 
 
 @Component 
 @StepScope 
 public class CustomReader extends FlatFileItemReader< CustomClass> {
 
 @Override 
 public void afterPropertiesSet（）throws Exception {
 setLineMapper（new DefaultLineMapper< CustomClass>（）{
 {
 /// configuration of line mapper 
} 
}）; 
 setRecordSeparatorPolicy（new BlankLineRecordSeparatorPolicy（））; 
 super.afterPropertiesSet（）; 
} 
} 
  

I am processing CSV files using FlatFileItemReader.

Sometimes I am getting blank lines within the input file.

When that happened the whole step stops. I want to skipped those lines and proceed normal.

I tried to add exception handler to the step in order to catch the execption instead of having the whole step stooped:

@Bean
    public Step processSnidUploadedFileStep() {
        return stepBuilderFactory.get("processSnidFileStep")
                .<MyDTO, MyDTO>chunk(numOfProcessingChunksPerFile) 
                .reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
                .processor(manualUploadAsyncItemProcessor())
                .writer(manualUploadAsyncItemWriter())
                .listener(logProcessListener)
                .throttleLimit(20)
                .taskExecutor(infrastructureConfigurationConfig.taskJobExecutor())
                .exceptionHandler((context, throwable) -> logger.error("Skipping record on file. cause="+ ((FlatFileParseException)throwable).getCause()))
                .build();
    }

Since I am processing with chunks when blank line arrives and exception is catched what's happens is that the whole chunk is skipped(the chunk might contains valid lines on CSV file and they are skipped aswell)

Any idea how to do this right when processing file in chunks?

Thanks, ray.

After editing my code. still not skipping:

public Step processSnidUploadedFileStep() {
        SimpleStepBuilder<MyDTO, MyDTO> builder = new SimpleStepBuilder<MyDTO, MyDTO>(stepBuilderFactory.get("processSnidFileStep"));
       return builder
                .<PushItemDTO, PushItemDTO>chunk(numOfProcessingChunksPerFile)
                .faultTolerant().skip(FlatFileParseException.class)
                .reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
                .processor(manualUploadAsyncItemProcessor())
                .writer(manualUploadAsyncItemWriter())
                .listener(logProcessListener)
                .throttleLimit(20)
                .taskExecutor(infrastructureConfigurationConfig.taskJobExecutor())
                .build();
    }

解决方案

We created custom SimpleRecordSeparatorPolicy which is telling reader to skip blank lines. That way we read 100 records, i.e. 3 are blank lines and those are ignored without exception and it writes 97 records.

Here is code:

package com.my.package;

import org.springframework.batch.item.file.separator.SimpleRecordSeparatorPolicy;

public class BlankLineRecordSeparatorPolicy extends SimpleRecordSeparatorPolicy {

    @Override
    public boolean isEndOfRecord(final String line) {
        return line.trim().length() != 0 && super.isEndOfRecord(line);
    }

    @Override
    public String postProcess(final String record) {
        if (record == null || record.trim().length() == 0) {
            return null;
        }
        return super.postProcess(record);
    }

}

And here is reader:

package com.my.package;

import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.stereotype.Component;

@Component
@StepScope
public class CustomReader extends FlatFileItemReader<CustomClass> {

    @Override
    public void afterPropertiesSet() throws Exception {
        setLineMapper(new DefaultLineMapper<CustomClass>() {
            {
                /// configuration of line mapper
            }
        });
        setRecordSeparatorPolicy(new BlankLineRecordSeparatorPolicy());
        super.afterPropertiesSet();
    }
}

这篇关于如何使用FlatFileItemReader和chunks在CSV中跳过空行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！