我们可以在一个批处理作业中拥有多个读取器(通过分页读取记录)吗? [英] Can we have multiple readers(reading records through pagination) in a single batch job

问题描述

我有一个春季批作业要写.其中:

I have a spring batch job to be written . where :

1. 我需要通过分页读取一次10k条记录(一次读取1000条记录)(从azure SQL db中读取)
2. 我需要一次使用1000条记录，然后使用这些记录的一列(比如说一些ID)从另一个cosmos db表中读取相应的记录.

在这种情况下，我该如何实现2个读取器，因为我需要一次读取1000条记录，然后再次从cosmos db中获取那1000条记录的记录并对其进行处理.

How do I implement 2 readers in this case as I need to read 1000 records at a time and again fetch records from cosmos db for those 1000 records first and process that.

推荐答案

有一个常见的模式称为，其中读者仅读取项目ID，而处理器执行另一个查询以获取项目详细信息.

There is a common pattern called the "driving query pattern" in which the reader reads only item IDs and a processor does another query to fetch item details.

此模式对于中小型数据集效果很好，但是您应该意识到，对于大型数据集，此模式效果不佳，因为每个项目都会有一个附加查询(也称为n + 1问题).

This pattern works well for small/medium data sets but you should be aware that it does not perform well for large datasets as there would be an additional query for each item (also known as the n+1 problem).

这篇关于我们可以在一个批处理作业中拥有多个读取器(通过分页读取记录)吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！