如何在不停机的情况下部署新作业 [英] how to deploy a new job without downtime

问题描述

我有一个 Apache Flink 应用程序，它从单个 Kafka 主题中读取数据.我想不时更新应用程序而不会遇到停机.目前，Flink 应用程序执行一些简单的操作，例如 map 和一些通过 http rest API 到外部系统的同步 IO.

I have an Apache Flink application that reads from a single Kafka topic. I would like to update the application from time to time without experiencing downtime. For now the Flink application executes some simple operators such as map and some synchronous IO to external systems via http rest APIs.

我尝试使用停止命令，但我收到作业终止 (STOP) 失败:此作业不可停止."，我了解 Kafka 连接器不支持停止行为 - 一个链接！一个简单的解决方案是使用保存点取消并使用保存点重新部署新 jar，但随后我们会停机.另一种解决方案是从外部控制部署，例如切换到新主题.

I have tried to use the stop command, but i get "Job termination (STOP) failed: This job is not stoppable.", I understand that the Kafka connector does not support the the stop behavior - a link! A simple solution would be to cancel with savepoint and to redeploy the new jar with the savepoint, but then we get downtime. Another solution would be to control the deployment from the outside, for example, by switching to a new topic.

什么是好的做法?

推荐答案

如果您不需要恰好一次的输出(即可以容忍一些重复)，您可以在不取消正在运行的作业的情况下获取保存点.保存点完成后，您将开始第二项工作.第二份工作可以写不同的主题，但不是必须的.当第二个作业完成时，您可以取消第一个作业.

If you don't need exactly-once output (i.e., can tolerate some duplicates) you can take a savepoint without cancelling the running job. Once the savepoint is completed, you start a second job. The second job could write to different topic but doesn't have to. When the second job is up, you can cancel the first job.

这篇关于如何在不停机的情况下部署新作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！