如何读取csv行chunked id列与Spring-Batch？ [英] How to read csv lines chunked by id-column with Spring-Batch?

问题描述

我使用 Spring-Batch 读取csv文件，格式化内容并将其写入数据库，如：

  StepBuilder< T，T> builder = stepBuilderFactory.get（step）
。< T，T> chunk（100）
 .reader（flatFileItemReader）
 .processor（processor）
 .writer jpaItemWriter）; 
  

csv包含ID列。如何修改阅读器以使该块基于该ID？
Examle：

  #id，#value 
 1，first 
 1000，second 
 1001，second 
 1005，second 
  

在这种情况下，

是否可以通过文件中的值应用分块？

解决方案

我也使用自定义 CompletionPolicy 和 PeekableItemReader 。
$
代码的背后的想法是窥探下一个项目，perfrom下一个元素读取，并从值更改中检查。
$
当值发生变化时，从 CompletionPolicy.isComplete（）返回 true >

重要事项：此政策必须注册为步骤监听者！

  public class BreakKeyCompletionPolicy extends CompletionPolicySupport {
 private BreakKeyCompletionContext cc; 
 private PeekableItemReader< Object>读者; 
 //用于检查价值中断的策略
 private BreakKeyStrategy< Object>战略; 
 
 public void setReader（PeekableItemReader< Object> forseeingReader）{
 this.reader = forseeingReader; 
} 
 
 @Override 
 public boolean isComplete（RepeatContext context）{
 return this.cc.isComplete（）; 
} 
 
 @Override 
 public RepeatContext start（RepeatContext context）{
 context.setAttribute（current，null）; 
 this.cc = new BreakKeyCompletionContext（context）; 
 return cc; 
} 
 / **上下文包含当前元素（current属性并管理下一个元素
 * Null下一个元素被视为键断点
 * / 
 protected class BreakKeyCompletionContext extends RepeatContextSupport {
 public BreakKeyCompletionContext（RepeatContext context）{
 super（context）; 
} 
 public boolean isComplete（）{
 final Object next; 
 try {
 next = reader.peek（）; 
} 
 catch（Exception e）{
 throw new NonTransientResourceException（Unable to peek，e）; 
} 
 if（null == next）{
 return true; 
} 
 return strategy.isKeyBreak（this.getAttribute（current），next）; 
} 
} 
 
 @AfterRead 
 public void afterRead（Object item）{
 this.cc.setAttribute（current，item）; 
} 
} 
  

I'm using Spring-Batch to read a csv file, format the content and write it to a database like:

StepBuilder<T, T> builder = stepBuilderFactory.get("step")
    .<T, T>chunk(100)
    .reader(flatFileItemReader)
    .processor(processor)
    .writer(jpaItemWriter);

The csv contains an ID column. How can I modify the reader to base the chunks on that ID? Examle:

#id, #value
1, first
1000, second
1001, second
1005, second

In this case the chunk would only read the first line, then commit, and then continue.

Is that possible to apply chunking by a value in the file?

解决方案

I did the same using a custom CompletionPolicy and a PeekableItemReader.
The idea behind the code is to peek next item, perfrom next element read and check from value change.
When a value change happens return true from CompletionPolicy.isComplete().

Important: this policy must be registered as step listener!

public class BreakKeyCompletionPolicy extends CompletionPolicySupport{
    private BreakKeyCompletionContext cc;
    private PeekableItemReader<Object> reader;
    // Strategy used to check for value break
    private BreakKeyStrategy<Object> strategy;

    public void setReader(PeekableItemReader<Object> forseeingReader){
        this.reader = forseeingReader;
    }

    @Override
    public boolean isComplete(RepeatContext context){
        return this.cc.isComplete();
    }

    @Override
    public RepeatContext start(RepeatContext context)   {
        context.setAttribute("current", null);
        this.cc = new BreakKeyCompletionContext(context);
        return cc;
    }
    /** Context contains current element ("current" property" and manage next element.
     * Null next element is treated as a key break
     */
    protected class BreakKeyCompletionContext extends RepeatContextSupport {
        public BreakKeyCompletionContext(RepeatContext context)     {
            super(context);
        }
        public boolean isComplete(){
            final Object next;
            try{
                next = reader.peek();
            }
            catch (Exception e){
                throw new NonTransientResourceException("Unable to peek", e);
            }
            if (null == next){
                return true;
            }
            return strategy.isKeyBreak(this.getAttribute("current"), next);
        }
    }

    @AfterRead
    public void afterRead(Object item){
        this.cc.setAttribute("current", item);
    }
}

这篇关于如何读取csv行chunked id列与Spring-Batch？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！