Spring Batch-读取字节流，进行处理，写入2个不同的csv文件，将其转换为输入流，并将其存储到ECS，然后写入数据库 [英] Spring Batch - Read a byte stream, process, write to 2 different csv files convert them to Input stream and store it to ECS and then write to Database

问题描述

我有一个要求，我们要通过ECS S3预签名url以字节流的形式接收一个csv文件.我必须验证数据并将验证成功和失败的记录写入2个不同的csv文件，然后通过将它们转换为InputStream将它们存储到ECS S3存储桶中.还将成功记录写入数据库，以及入站，成功和失败文件的预签名URL.

I have a requirement where we receive a csv file in the form of byte stream through ECS S3 Pre-Signed url. I have to validate the data and write the validation successful and failed records to 2 different csv files and store them to ECS S3 bucket by converting them to InputStream. Also write the successful records to database and also the pre-signed urls of the inbound, success and failure files.

我是Spring Batch的新手.我应该如何满足此要求?

I'm new to Spring Batch. How should I approach this requirement?

如果我选择一个FlatFileItemReader来读取，则使用ItemProcessor来处理数据，我该如何写入不同的文件和数据库中?

If I choose a FlatFileItemReader to read, ItemProcessor to process the data how should I write to different files and to Database?

或

我应该使用Tasklets创建工作吗?TIA.

Should I create a job using Tasklets? TIA.

推荐答案

请在下面找到示例代码片段.让我知道您是否遇到任何问题

Please find below sample code snippet . Let me know if you face any issues

 //Your InputOutPut DTO This is the key object
   Class BaseCSVDTO {
    // yourCSVMappedFields  
    private SuccessCSVObject successObject;
    private FailureCSVObject failureObject;
   }

   //Read the Files in reader as Normal better create a custom reader if you want to get more control
    @Bean
    public ItemReader<BaseCSVDTO> yourFlatFileItemReader() {
        
         //populate mapped fields automatically by Springbatch
    }

    @Bean
    public CSVProcessor csvValidationProcessor() {
        return new CSVProcessor();
    }
    
    Class CSVProcessor implements ItemProcessor<BaseCSVDTO, BaseCSVDTO> {
        @Override
        public BaseCSVDTO CSVProcessor(BaseCSVDTO eachCSVitem) throws Exception {
            //validateEachItem and put in Success or Failure Object
            //Example of Success
                SuccessCSVObject successObject = new SuccessCSVObject()
                eachCSVitem.setSuccessObject(successObject);
            //Same way for Failure object   
        }
    }

   @Bean
    public CompositeItemWriter compositeWriter() throws Exception {
        CompositeItemWriter compositeItemWriter = new CompositeItemWriter();
        List<ItemWriter> writers = new ArrayList<ItemWriter>();
        writers.add(successCSVWriter());
        writers.add(failureCSVWriter());
        compositeItemWriter.setDelegates(writers);
        return compositeItemWriter;
    }

    @Bean
    public YourItemWriter<BaseCSVDTO> successCSVWriter() {
        return new SuccessWriter();
    }

    @Bean
    public YourItemWriter<BaseCSVDTO> failureCSVWriter() {
        return new FailureWriter;
    }

    
    public class SuccessWriter implements ItemWriter<BaseCSVDTO> {
        @Override
        public void write(List<? extends BaseCSVDTO> items){
        for(BaseCSVDTO baseCSVDTO:items) {
            baseCSVDTO.getSuccessObject
          //write Success CSV 
        }
        }
    }

  public class FailureWriter implements ItemWriter<BaseCSVDTO> {
        @Override
        public void write(List<? extends BaseCSVDTO> items){
        for(BaseCSVDTO baseCSVDTO:items) {
          //write Success CSV 
          baseCSVDTO.getFailureObject
        }
        }
    }

    /// Finally Job step
    @Bean
    public Step executionStep() throws Exception {
        return stepBuilderFactory.get("executionStep").<BaseCSVDTO, BaseCSVDTO>chunk(chunkSize)
                .reader(yourFlatFileItemReader()).processor(csvValidationProcessor()).writer(compositeWriter())
                //.faultTolerant()
                //.skipLimit(skipErrorCount).skip(Exception.class)//.noSkip(FileNotFoundException.class)
                //.listener(validationListener())
                //.noRetry(Exception.class)
                //.noRollback(Exception.class)
                .build();
    }

这篇关于Spring Batch-读取字节流，进行处理，写入2个不同的csv文件，将其转换为输入流，并将其存储到ECS，然后写入数据库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！