Kubernetes 上的批处理 [英] Batch Processing on Kubernetes
问题描述
这里有没有人在 kubernetes 上有过批处理(例如 spring 批处理)的经验?这是个好主意吗?如果我们使用 kubernetes 自动缩放功能，如何防止批处理处理相同的数据?谢谢.
Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ? How to prevent batch processing process same data if we use kubernetes auto scaling feature ? Thank you.
推荐答案
这里有没有人在 kubernetes 上有过批处理(例如 spring 批处理)的经验?这是个好主意吗?
Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ?
对于 Spring Batch，我们(Spring Batch 团队)在这方面确实有一些经验，我们在以下会谈中分享:
For Spring Batch, we (the Spring Batch team) do have some experience on the matter which we share in the following talks:
- Kubernetes 上的云原生批处理，作者:Michael Minella
- Kubernetes 上的 Spring Batch，由我编写.
- Cloud Native Batch Processing on Kubernetes, by Michael Minella
- Spring Batch on Kubernetes, by me.
在 Kubernetes 上运行批处理作业可能很棘手:
Running batch jobs on kubernetes can be tricky:
- pod 可能会在处理过程中被 k8s 重新调度到不同的节点
- cron 作业可能被触发两次
- 等
这需要开发人员进行额外的非平凡工作，以确保批处理应用程序具有容错性(对节点故障、Pod 重新调度等具有弹性)并且可以防止在集群环境中重复执行作业.
This requires additional non-trivial work on the developer's side to make sure the batch application is fault-tolerant (resilient to node failure, pod re-scheduling, etc) and safe against duplicate job execution in a clustered environment.
Spring Batch 会为您处理这些额外的工作，出于以下几个原因，它可以成为在 k8s 上运行批处理工作负载的不错选择:
Spring Batch takes care of this additional work for you and can be a good choice to run batch workloads on k8s for several reasons:
- 成本效率:Spring Batch 作业在外部数据库中维护它们的状态，这使得在作业/节点故障的情况下可以从最后一个保存点重新启动它们或 pod 重新调度
- 稳健性:得益于集中式作业存储库，可防止重复作业执行
- 容错:重试/跳过失败的项目，以防出现暂时性错误，例如调用可能暂时关闭或在云环境中重新安排的网络服务
- Cost efficiency: Spring Batch jobs maintain their state in an external database, which makes it possible to restart them from the last save point in case of job/node failure or pod re-scheduling
- Robustness: Safe against duplicate job executions thanks to a centralized job repository
- Fault-tolerance: Retry/Skip failed items in case of transient errors like a call to a web service that might be temporarily down or being re-scheduled in a cloud environment
我写了一篇博客文章，其中我用代码示例详细解释了所有这些方面.你可以在这里找到它:Kubernetes 上的 Spring Batch:大规模高效批处理
I wrote a blog post in which I explain all these aspects in details with code examples. You can find it here: Spring Batch on Kubernetes: Efficient batch processing at scale
如果我们使用kubernetes自动缩放功能，如何防止批处理处理相同的数据?
How to prevent batch processing process same data if we use kubernetes auto scaling feature ?
使每个作业处理不同的数据集是可行的方法(例如，每个文件一个作业).但是您可能对不同的模式感兴趣，请参阅 工作模式 来自 k8s 文档.
Making each job process a different data set is the way to go (a job per file for example). But there are different patterns that you might be interested in, see Job Patterns from k8s docs.
这篇关于Kubernetes 上的批处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
