如何使用Spring批处理有效且正确地将顺序活动加载到数据库? [英] How to effectively and correctly load sequential activities to Database using Spring batch?

问题描述

我目前正在一个项目中，将.dat文件信息加载到数据库中.但是，此.dat文件不仅包含数据，而且还包含操作.第一个字段指示记录的操作，所有其他字段仅是数据. 以下是一些示例记录: 一键1键2数据1数据2 D键1键2数据1数据2 C key1 key2 data1 data2

I'm currently working on a project loading a .dat file info to database. However, this .dat file contains not only data but also actions. The first field indicates the action of the records and all else are just data. Below are some examples records: A key1 key2 data1 data2 D key1 key2 data1 data2 C key1 key2 data1 data2

let，A =添加，D =删除，C =更新 文件大小约为5GB. 在这种情况下，要处理的记录的顺序确实很重要.可以使用Spring批处理对此进行批处理吗?我的前任所做的以前的实现是创建3个列表:addList，deleteList，updateList，并分别为它们生成dml语句.虽然在加载时非常高效，但由于执行这些列表的顺序无关紧要，因此无法正常工作，但可能会导致数据不一致.因此，为了保持数据的有效性，我相信这些记录必须按顺序执行.

let, A=add, D=delete, C=update The file size is roughly 5GB. In this case, order of the records to be process does matter. Is it possible to use Spring batch to batch process this? A previous implementation done by my predecessor was to create 3 lists: addList, deleteList, updateList, and generate dml statements for them separately. Although very efficient when loading, but it doesn't work because doesn't matter which order you execute these lists, it might cause data inconsistency. So to keep the data valid, I believe these records have to be executed in order.

我想出的一个解决方案是将读取器的读取大小设置为1.但这似乎违背了spring batch的目的.

One solution I came up with is to set reader fetch-size to 1. But that seems to defeat the purpose of spring batch.

是否有更好的方法可以做到这一点?也许不是春季批处理，而是其他?

Are there better ways to do this? perhaps not spring batch but something else?

推荐答案

您应该能够(通过保留记录顺序)使用以下组合实现此目标:

You should be able to achieve this (while preserving records order) using a combination of:

• PatternMatchingCompositeLineMapper:根据模式(A，D和C)映射项目
• ClassifierCompositeItemWriter:配置为PatternMatchingClassifier.这将根据项目的类型对项目进行分类，并使用相应的作者(每种类型应有一位作者)来执行操作.

• PatternMatchingCompositeLineMapper: to map items according to your patterns (A, D and C)
• ClassifierCompositeItemWriter: configured with a PatternMatchingClassifier. This will classify items according to their type and use the corresponding writer (you should have one writer per type) to do the action.

这篇关于如何使用Spring批处理有效且正确地将顺序活动加载到数据库?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！