Spring批处理，读取器如何工作？如果结果集发生变化？ [英] Spring batch chunk processing , how does the reader work ?if the result set changes?

问题描述

我是springBatch分块的新手。我想了解阅读器的工作原理

I'm new to springBatch chunking. I want to understand how reader works

这里是这种情况：实现清除用户帐户
块处理器：
有一个读取所有内容的阅读器与清除条件匹配的用户帐户（顺序）。
处理器：根据一些计算为每个用户帐户，它可能会创建一个新的用户帐户并更改当前记录（例如将其标记为已清除）

here is the scenario : implementing a purging of user accounts Chunk processor : have a reader which reads all the user accounts that matches with purge criteria ,in an order. processor : for each user account based on the some calculation ,it may create a new user account and also changes current record(say mark it as purged)

问题：读者如何工作？说我有5000个用户帐户。如果我的块大小为1000

question : how doe the reader work? say i have 5000 user accounts. If my chunk size is 1000

读者将读取1000条记录，然后启动处理器。 （例如，处理器创建了另外100条新记录），现在写者写入更新的记录

will reader reads 1000 records and then starts processor . (say processor creates another 100 new records ) ,now writer writes whatever records updated

以读取接下来的1000条记录，读取器将再次执行查询吗？

for reading next 1000 records will the reader executes query again? how does it know where to start?

我正在使用休眠模式。

推荐答案

要回答您的特定问题，取决于您使用的 ItemReader 实现。如果您使用的是 JdbcCursorItemReader ，则在整个过程中我们都将光标保持打开状态，因此实际上是从执行一个查询中读取。如果您使用的是 JdbcPagingItemReader ，则下一个块的开始位置基于分页逻辑。

To answer your specific question, it depends on the ItemReader implementation you use. If you're using the JdbcCursorItemReader, we hold the cursor open during the entire process so we're really reading from the execution of one query. If you're using the JdbcPagingItemReader, then where the next chunk begins is based on the pagination logic.

A几个注意事项：

1. 使用Hibernate进行批处理可能很棘手。使用Hibernate时还增加了一些复杂性，可以避免直接进入数据库时​​的麻烦（更不用说在批处理环境中可能带来的性能优势）。


2. 请记住，Spring Batch不提供检查是否基础数据集已更改。如果您使用的是 JdbcPagingItemReader ，则每个查询都是唯一查询，因此，如果您添加符合条件的记录，它们也会被返回（我不是100％确保在打开游标时更改基础数据会发生什么……这可能是数据库本身的功能）。通常，您会使用一些from标记（时间戳，处理标记等）来标记要在该批处理中处理的记录。

1. Using Hibernate can be tricky with batch processing. There are added complexities when using Hibernate that you can avoid when going straight to the database (not to mention potential performance benefits in a batch environment).
2. Keep in mind that Spring Batch provides no checks for if the underlying dataset has changed. If you're using the JdbcPagingItemReader, each query is a unique query so if you add records that meet the criteria, they will be returned as well (I'm not 100% sure what would happen if the underlying data changed while a cursor was open…it may be a function of the db itself). Typically, you'll tag the records you want to process in that batch run with some from of flag (timestamp, processing flag, etc).

这篇关于Spring批处理，读取器如何工作？如果结果集发生变化？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！