Spring MongoItemReader 不会在单次执行时读取所有记录 [英] Spring MongoItemReader not reading all records on single execution

问题描述

@Bean
public Job orderJob() throws Exception {
    return jobBuilderFactory.get("orderJob").incrementer(new RunIdIncrementer()).listener(listener())
            .flow(orderStep()).end().build();
}

@Bean
public Step orderStep() throws Exception {
    return stepBuilderFactory.get("orderStep").<OrderCollection, Order>chunk(1000)
            .reader(orderReader()).processor(orderProcessor()).writer(orderWriter())
            .allowStartIfComplete(true).build();

}
@Bean
@StepScope
public MongoItemReader<OrderCollection> orderReader() throws Exception {
    MongoItemReader<OrderCollection> reader = new MongoItemReader<>();
    reader.setTemplate(mongoTemplate);
    reader.setCollection("order");
    Map<String, Sort.Direction> sort = new HashMap<>();
    sort.put("_id", Sort.Direction.ASC);
    reader.setSort(sort);
    reader.setTargetType(OrderCollection.class);
    reader.setQuery("{$or: [ {flag:false}, {flag:null} ]}");
    return reader;
}
@Bean
@StepScope
public OrderProcessor orderProcessor() {
    return new OrderProcessor();
}

@Bean
@StepScope
public ItemWriter<Order> orderWriter() {
    return new OrderWriter();
}

有序集合有5686条记录，所有记录如果为false则标志.但读取器在第一次运行时只读取和处理3000条记录.第二次运行 1686 条记录，第三次运行 1000 条记录.没有错误仅供参考

There are 5686 records in order Collection and for all records the flag if false .But the reader reads and process only 3000 records in first run. 1686 records in second run , and 1000 records in third run. There is no error FYI

推荐答案

我猜您可能正在更新您阅读的集合，并且您也在更新查询正在使用的字段.如果是这样，那么我最近也遇到了同样的问题.

I'm guessing that you're probably updating the collection your reading from and your also updating a field that the query is using. If so then I had the same problem recently.

MongoItemReader 是一个分页阅读器.因此，每次写入者更新这些记录时，读取者的池较小，但页面仍在增加.

The MongoItemReader is a paged reader. So each time the writer updates those records the reader has a smaller pool but the page is still increasing.

假设我们有 20 个项目并且一次阅读 5 个项目:

So imagine that we have 20 items and read 5 items at a time:

1) 从总共 20 项中读取 1-5 项.

1) Reads Items 1-5 from a total of 20.

2) 更新项目 1-5，现在总共有 15 个可能的项目

2) Updates Items 1-5 and now there's a total of 15 possible items

3) 从总共 15 个项目中读取 6-10 个项目.

3) Reads items 6-10 from a total 15.

4) 更新了 6-10 项，现在总共有 10 个可能的项.

4) Updates items 6-10 and now there's a total of 10 possible items.

5) 读取 10 个可能的项目中的第 11-15 个

5) Reads items 11-15 of 10 possible items

6) Read 返回 null，因为该页面没有返回任何内容.

6) Read returns null because there's nothing returned for that page.

所以现在您只处理了一半.

So now you've only processed half.

我按照下面的教程创建了一个 MongoDbCursorItemReader，它为我解决了这个问题:https://blog.zenika.com/2012/05/23/spring-batch-and-mongodb-cursor-based-item-reader/

I followed the tutorial below to create a MongoDbCursorItemReader which solved this problem for me: https://blog.zenika.com/2012/05/23/spring-batch-and-mongodb-cursor-based-item-reader/

这篇关于Spring MongoItemReader 不会在单次执行时读取所有记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！