Flink动态缩放 [英] Flink dynamic scaling

问题描述

我目前正在研究Flink的可伸缩性.从版本1.2.0开始，引入了动态重缩放.我正在考虑扩展一项长期运行的工作，该工作将从Kafka源读取数据.

I am currently studying scalability on Flink. Starting from Version 1.2.0, dynamic rescaling was introduced. I am looking at scaling a long running job which reads data from Kafka source.

有关动态缩放的问题.

1. 要扩展我的flink应用程序，例如:添加新的任务管理器，是否必须重新启动作业/yarn会话才能使用新添加的资源?
2. 我认为可以编写Yarn客户端来部署新的任务管理器并使其与作业管理器对话，这已经可以在现有的flink yarn客户端应用程序中使用吗?

如果这些问题太基本了，请原谅我，我确实浏览了文档，并且不得不承认我最近还无法将这些概念与对纱线的一些测试部署完全结合在一起.

Pardon me if these questions are too basic, I did go through the documents and I have to admit I have not been able to put the concepts altogether with some test deployments on yarn recently.

推荐答案

当前，动态缩放是指针对键控状态或非键控状态更新操作员并行性(Flink 1.2)的能力.键控状态.

Currently, Dynamic Scaling means the capability to update the operators' parallelism(Flink 1.2), either for keyed state or for non-keyed state.

1. 要扩展我的flink应用程序，例如:添加新的任务管理器，我是否必须重新启动作业/yarn会话才能使用新添加的资源?-是，必须首先停止作业，更新并行性，然后再次重新启动它.不必担心状态，Flink会处理它们，包括重新分区.

1. To scale out my flink application, for example: add new task managers, must I restart the job / yarn session to use the newly added resource? - Yes, the job has to be stopped first, update the parallelism, and restart it again. Do not have to worry about the state, Flink will handle them, including repartition.

我认为可以编写Yarn客户端来部署新的任务管理器并使其与作业管理器对话，这在现有的flink yarn客户端应用程序中已经可用吗?-否，您不能.将来似乎会添加此功能.目前，我们无法做到这一点.

I think it's possible to write Yarn client to deploy new task managers and make it talk to job manager, is that already available in existing flink yarn client application? - No, you can not. This feature seems to be added in the future. Currently, we can not do that.

这篇关于Flink动态缩放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！