我应该如何使用 Spring Batch 和 Spring Cloud Data Flow 切片和编排可配置的批处理网络? [英] How should I slice and orchestrate a configurable batch network using Spring Batch and Spring Cloud Data Flow?
问题描述
我们希望将一些 Kettle 导入作业的调度和顺序控制从专有实现迁移到 Spring Batch 风格的良好实践实现.
We would like to migrate the scheduling and sequence control of some Kettle import jobs from a proprietary implementation to a Spring Batch flavour, good practice implementation.
我打算使用 Spring Cloud Data Flow (SCDF) 服务器来实现和运行现有外部导入作业的可配置序列.
I intend to use Spring Cloud Data Flow (SCDF) server to implement and run a configurable sequence of the existing external import jobs.
SCDF 控制台任务编辑器 UI 似乎有望组装一个流程.所以一个 Task 包装了一个 Spring Batch，它在一个步骤中只执行一个 Tasklet，启动和轮询 Carte REST API.到目前为止，这是否有意义?
The SCDF console Task editor UI seems promising to assemble a flow. So one Task wraps one Spring Batch, which in a single step only executes a Tasklet starting and polling the Carte REST API. Does this make sense so far?
您有更好的实施建议吗?
Would you suggest a better implementation?
限制和要求:
- 使用 Carte REST API 触发和轮询外部 Kettle 作业.实际上，这是一个单独的 Kettle 作业实现，为每个要导入的实体使用单独的参数调用.
- 有多个实体的导入作业的可配置有向图，其中一些依赖于先前实体类型的正确导入.(例如部门，然后是员工，然后是角色分配...)
- 随着即将到来的实施，我们希望得到
- 监视和控制(开始、中止、暂停、恢复)
- 可重启性
- 在生产中轻松重新配置序列(可能通过 GUI 或外部编辑器)
- 可能是一些报告和统计数据.
据我目前的理解，这可以通过使用 Spring Cloud Data Flow (SCDF) 服务器和一些 Task/Batch 实现/组合来实现.
As my current understanding, this could be achieved by using Spring Cloud Data Flow (SCDF) server, and some Task / Batch implementation / combination.
如果我错了，请纠正我，但是带有硬连线流程的单个 Spring Batch 作业似乎不太适合我.或者是否有一种简单的方法来编辑和重新部署具有更改生产流程的 Spring Batch?我找不到任何东西，甚至找不到用于批处理 XML 表示的易于使用的编辑器.
Correct me if I'm wrong, but a single Spring Batch job with its hardwired flow seems not very suitable to me. Or is there an easy way to edit and redeploy a Spring Batch with changed flow in production? I couldn't find anything, not even an easy to use editor for the XML representation of a batch.
推荐答案
是的，我相信您可以使用 Spring Cloud Data Flow 和 Spring Cloud Task/Spring Batch 来实现您的设计目标.
Yes, I believe you can achieve your design goals using Spring Cloud Data Flow along with the Spring Cloud Task/Spring Batch.
正如您从另一个 SO 线程指出的那样，可以使用 Spring Cloud Data Flow 管理多个 Spring Batch 作业(使用组合任务)的流程.
The flow of multiple Spring Batch Jobs (using the Composed Task) can be managed using Spring Cloud Data Flow as you pointed from the other SO thread.
使用 Carte REST API 触发和轮询外部 Kettle 作业.实际上，这是一个单独的 Kettle 作业实现，为每个要导入的实体使用单独的参数调用.
The external Kettle jobs are triggered and polled using Carte REST API. Actually, it's one single Kettle job implementation, called with individual parameters for each entity to be imported.
有多个实体的导入作业的可配置有向图，其中一些依赖于先前实体类型的正确导入.(例如部门，然后是员工，然后是角色分配...)
There is a configurable, directed graph of import jobs for several entities, some of them being dependent on a correct import of the previous entity type. (e.g. Department, then Employee, then Role assignments...)
同样，以上两者都可以作为组合任务进行管理(组合任务由常规任务和基于 Spring Batch 的应用程序组成).
Again, both the above can be managed as a Composed Task (with the composed task consisting of a regular task as well as Spring Batch based applications).
您可以在调用时通过批处理作业参数或任务/批处理应用程序属性或简单的命令行参数管理传递给每个任务/批处理的参数.
You can manage the parameters passed to each task/batch upon invocation via batch job parameters or task/batch application properties or simply command-line arguments.
随着即将到来的实施，我们希望得到监视和控制(启动、中止、暂停、恢复)可重启性生产中序列的轻松重新配置(可能通过 GUI 或外部编辑器)可能有一些报告和统计数据.
With the upcoming implementation, we would like to get monitoring and controlling (start, abort, pause, resume) restartability easy reconfigurability of the sequence in production (possibly by GUI, or external editor) possibly some reporting and statistics.
Spring Cloud Data Flow 可帮助您实现这些目标.您可以访问任务开发人员指南 和 任务监控指南 了解更多信息.
Spring Cloud Data Flow helps you achieve these goads. You can visit the Task Developer Guide and the Task Monitoring Guide for more info.
您还可以从网站上查看批量开发人员指南嗯.
You can also check the Batch developer guide from the site as well.
这篇关于我应该如何使用 Spring Batch 和 Spring Cloud Data Flow 切片和编排可配置的批处理网络?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
