如何汇总Spring Batch作业中的完整数据集? [英] Howto aggregate on full data set in Spring Batch jobs?

问题描述

我需要在Spring Batch作业中插入聚合. 但是聚合步骤需要使整个数据集可用.

I need to insert aggregation in my Spring Batch jobs. But the aggregation step need to have the entire data set available.

在纯SQL中，编写SQL聚合请求很容易:完整的数据集(存储在数据库中)是可用的.

In pure SQL, it's easy to code SQL aggregation requests : the full data set (as stored in database) is available.

但是在Spring Batch作业中，所有事情都在内存中完成，并分块传播. 那么如何处理这类数据呢?

But in Spring Batch jobs, everything is done in memory, and spread in chunked. So howto deal with that kind of data strewing ?

您是否对插入聚合步骤/过程的最佳实践有任何建议?

Do you have any advice concerning the best practices to insert aggregation steps/processes ?

非常感谢您的启迪

推荐答案

您有分区选项，可以具有StepExecutionAggregator，它具有聚合方法，该方法接受所有分区步骤的StepContext列表.

You have Partitioning option in spring batch which can have StepExecutionAggregator, it has aggregate method which accepts list of StepContext of all partitioned steps.

我们已经与肥皂服务器集成，在这里我们首先收到需要处理的东西的列表，然后将其划分为子步骤并并行处理，并在每个子步骤完成后调用聚合器，该聚合器可以根据数据进行处理在子步骤的上下文中.

We had i.e. integration with soap server where we first received list of something that needs to be processed, than we partitioned it to child steps and processed in parallel and after each child step finishes aggregator is invoked which can do stuff based on data in child step context.

如果数据中有某些东西可以很好地进行分区(例如从数据库中提取项目列表并并行处理每个项目，在步骤上下文中保存项目数据，使用聚合器并合并每个项目中的所有内容)，这是一个好方法步骤上下文并对组合数据进行通用操作.

It is good way if you have something in your data which can be good rule for partitioning (i.e. pull list of items from DB and process each item in parallel, save item data in step context, use aggregator and combine everything in each step context and do common operation on combined data).

在此处链接可以进行分区示例(有不是聚合，但您可以将其添加到masterStep).

Here is link to example with partitioning (there is no aggregation but you can add it to masterStep).

这篇关于如何汇总Spring Batch作业中的完整数据集?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！