Flink 动态缩放 [英] Flink dynamic scaling
问题描述
我目前正在研究 Flink 的可扩展性.从版本 1.2.0 开始，引入了动态重新缩放.我正在考虑扩展从 Kafka 源读取数据的长时间运行的作业.
I am currently studying scalability on Flink. Starting from Version 1.2.0, dynamic rescaling was introduced. I am looking at scaling a long running job which reads data from Kafka source.
关于动态缩放的问题.
- 要横向扩展我的 flink 应用程序，例如:添加新的任务管理器，我是否必须重新启动作业/yarn 会话才能使用新添加的资源?
- 我认为可以编写 Yarn 客户端来部署新的任务管理器并使其与作业管理器通信，现有的 flink 纱线客户端应用程序中是否已经提供了该应用程序?
请原谅我，如果这些问题太基本了，我确实浏览了文档，我不得不承认我最近无法将这些概念与纱线上的一些测试部署完全结合起来.
Pardon me if these questions are too basic, I did go through the documents and I have to admit I have not been able to put the concepts altogether with some test deployments on yarn recently.
推荐答案
目前，Dynamic Scaling 意味着更新算子并行性的能力(Flink 1.2)，无论是键控状态还是非-键控状态.
Currently, Dynamic Scaling means the capability to update the operators' parallelism(Flink 1.2), either for keyed state or for non-keyed state.
要扩展我的 flink 应用程序，例如:添加新的任务管理器，我是否必须重新启动作业/纱线会话才能使用新添加的资源? - 是，必须先停止作业，更新并行度，然后再次重新启动.不用担心状态，Flink 会处理，包括重新分区.
To scale out my flink application, for example: add new task managers, must I restart the job / yarn session to use the newly added resource? - Yes, the job has to be stopped first, update the parallelism, and restart it again. Do not have to worry about the state, Flink will handle them, including repartition.
我认为可以编写 Yarn 客户端来部署新的任务管理器并使其与作业管理器通信，现有的 flink 纱线客户端应用程序中是否已经提供了? - 否，你不能.此功能似乎在未来添加.目前，我们不能这样做.
I think it's possible to write Yarn client to deploy new task managers and make it talk to job manager, is that already available in existing flink yarn client application? - No, you can not. This feature seems to be added in the future. Currently, we can not do that.
这篇关于Flink 动态缩放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
