Spring批处理汇总值并写入单个值 [英] Spring batch to aggregate values and write single value
问题描述
我正在使用Spring Batch，我需要实现以下目标
- 读取csv文件，其中包含日期和金额等详细信息
- 汇总同一日期所有金额的总和
- 坚持一个带有日期和总和的条目
我过去使用过批处理，因此想到了以下方法.创建一个包含两个步骤的批处理.
第1步:
- 阅读器:使用FlatFileItemReader遍历整个文件
- 处理器:使用键作为日期和值作为数量填充地图.如果存在条目，则获取值并将其添加到新值
- 作家:没有操作作家，因为我不想写作
第2步:
- 阅读器:遍历地图的值
- 作家:坚持价值观
我能够完成第一步Map.此Map已用@JobScope
我被困在如何为第2步创建读取器的过程中，该读取器仅需要读取值列表.我尝试了ListItemReader，但无法从ListItemReader访问Map.
请提供解决方案或您有更好的方法来解决此问题
谢谢
选项1: 如果您的简历已经按日期排序，则可以实现一个组读取器，该读取器读取行直到键值更改为止.之后，整个组可以作为一项传递给处理器.
这样的小组阅读器可能看起来像这样:
private SingleItemPeekableItemReader<I> reader;
private ItemReader<I> peekReaderDelegate;
@Override
public void afterPropertiesSet() throws Exception {
Assert.notNull(peekReaderDelegate, "The 'itemReader' may not be null");
this.reader= new SingleItemPeekableItemReader<I>();
this.reader.setDelegate(peekReaderDelegate);
}
@Override
// GroupDTO is just a simple container. It is also possible to use
// List<I> instead of GroupDTO<I>
public GroupDTO<I> read() throws Exception {
State state = State.NEW; // a simple enum with the states NEW, READING, and COMPLETE
GroupDTO<I> group = null;
I item = null;
while (state != State.COMPLETE) {
item = reader.read();
switch (state) {
case NEW: {
if (item == null) {
// end reached
state = State.COMPLETE;
break;
}
group = new GroupDTO<I>();
group.addItem(item);
state = State.READING;
I nextItem = reader.peek();
// isGroupBreak returns true, if 'item' and 'nextItem' do NOT belong to the same group
if (nextItem == null || getGroupBreakStrategy.isGroupBreak(item, nextItem)) {
state = State.COMPLETE;
}
break;
}
case READING: {
group.addItem(item);
// peek and check if there the peeked entry has a new date
I nextItem = peekEntry();
// isGroupBreak returns true, if 'item' and 'nextItem' do NOT belong to the same group
if (nextItem == null || getGroupBreakStrategy.isGroupBreak(item, nextItem)) {
state = State.COMPLETE;
}
break;
}
default: {
throw new org.springframework.expression.ParseException(groupCounter, "ParsingError: Reader is in an invalid state");
}
}
}
return group;
}
您需要一个SingleItemPeekableItemReader，以便预读下一个元素.这个包装了您的普通读者.
选项2: 步骤1如您所建议，但是只需为步骤2编写一个tasklet.不需要使用reader-process-writer方法，而是可以使用一个简单的tasklet将地图内容写入文件. >
选项3: 如果您真的想在第2步中使用阅读器-处理器-书写器方法，请编写自己的遍历地图的阅读器.
类似的东西(我没有测试该代码):
public class MapReader implements ItemReader {
private MapContainer container;
private Iterator<Map.Entry<Date, Integer> mapIterator;
@PostConstruct
public void afterPropertiesSet() {
Assert.notNull(container);
iterator = container.getMap().entry().iterator;
}
public void setMapContainer(MapContainer container) {
this.container = container;
}
public Map.Entry<Date, Integer> read() {
if (iterator.hasNext()) {
return iterator.next();
}
return null;
}
}
@Component
public class MapContainer {
private Map<Date, Integer> data = new Hashmap<>();
public Map<Date, Integer> getMap() {
return data;
}
// add modifier method as needed for step 1
}
因此，您为Container创建了一个单个spring-bean实例，将其注入到第2步的处理器中，在其中填充它，还将其注入到上面的阅读器中.
I am using spring batch and I need to achieve the following
- Read a csv file which has details like date and amount
- Aggregate the sum of all amounts for a same date
- Persist one entry with date and the sum
I have used batch in the past and I thought of the following approach. Create a batch with 2 steps.
Step 1:
- Reader: Loop through the entire file using FlatFileItemReader
- Processor: Populate a map with Key as date and value as amount. If entry is present then get the value and add it to the new value
- Writer: No operation writer as I do not wish to write
Step 2:
- Reader: Loop through the values of the map
- Writer: Persist the values
I was able to acheive step 1 where I populated the Map. This Map has been declared with @JobScope
I am stuck at how do I create the reader for step2 which needs to just read the List of values. I tried ListItemReader but I am not able to access the Map from the ListItemReader.
Please advise a solution or if you have a better approach to tackle this
Thanks
Option 1: If your cvs is already sorted by date, you could implement a group reader, which reads lines until a key value changes. After that, the whole group can be passed as one item to the processor.
Such a group reader could look like this:
private SingleItemPeekableItemReader<I> reader;
private ItemReader<I> peekReaderDelegate;
@Override
public void afterPropertiesSet() throws Exception {
Assert.notNull(peekReaderDelegate, "The 'itemReader' may not be null");
this.reader= new SingleItemPeekableItemReader<I>();
this.reader.setDelegate(peekReaderDelegate);
}
@Override
// GroupDTO is just a simple container. It is also possible to use
// List<I> instead of GroupDTO<I>
public GroupDTO<I> read() throws Exception {
State state = State.NEW; // a simple enum with the states NEW, READING, and COMPLETE
GroupDTO<I> group = null;
I item = null;
while (state != State.COMPLETE) {
item = reader.read();
switch (state) {
case NEW: {
if (item == null) {
// end reached
state = State.COMPLETE;
break;
}
group = new GroupDTO<I>();
group.addItem(item);
state = State.READING;
I nextItem = reader.peek();
// isGroupBreak returns true, if 'item' and 'nextItem' do NOT belong to the same group
if (nextItem == null || getGroupBreakStrategy.isGroupBreak(item, nextItem)) {
state = State.COMPLETE;
}
break;
}
case READING: {
group.addItem(item);
// peek and check if there the peeked entry has a new date
I nextItem = peekEntry();
// isGroupBreak returns true, if 'item' and 'nextItem' do NOT belong to the same group
if (nextItem == null || getGroupBreakStrategy.isGroupBreak(item, nextItem)) {
state = State.COMPLETE;
}
break;
}
default: {
throw new org.springframework.expression.ParseException(groupCounter, "ParsingError: Reader is in an invalid state");
}
}
}
return group;
}
You need a SingleItemPeekableItemReader, in order to pre-read the next element. This one wraps your normal reader.
Option 2: Step one is as you have proposed, but simply write a tasklet for step 2. There is no need to use reader-process-writer approach, instead a simple tasklet could be used that writes the content of your map to a file.
Option 3: If you really wanna use a reader-processor-writer approach for step 2, write your own reader that iterates over your map.
something like (I did not test that code):
public class MapReader implements ItemReader {
private MapContainer container;
private Iterator<Map.Entry<Date, Integer> mapIterator;
@PostConstruct
public void afterPropertiesSet() {
Assert.notNull(container);
iterator = container.getMap().entry().iterator;
}
public void setMapContainer(MapContainer container) {
this.container = container;
}
public Map.Entry<Date, Integer> read() {
if (iterator.hasNext()) {
return iterator.next();
}
return null;
}
}
@Component
public class MapContainer {
private Map<Date, Integer> data = new Hashmap<>();
public Map<Date, Integer> getMap() {
return data;
}
// add modifier method as needed for step 1
}
so, you create a single spring-bean instance for the Container, inject it in your processor of step 2, fill it there, also inject it in the reader above.
这篇关于Spring批处理汇总值并写入单个值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
