在 YARN 集群中提交作业之前，Flink TaskManager 不会启动 [英] Flink TaskManagers do not start until job is submitted in YARN cluster

问题描述

我正在使用 Amazon EMR 在 YARN 上运行 Flink 集群.我的设置包括用于 1 个主节点和 2 个核心节点的 m4.large 实例.我已经使用以下命令在 YARN 上启动了 Flink CLuster:flink-yarn-session -n 2 -d -tm 4096 -s 4.

I am using Amazon EMR to run Flink Cluster on YARN. My setup consists of m4.large instances for 1 master and 2 core nodes. I have started the Flink CLuster on YARN with the command: flink-yarn-session -n 2 -d -tm 4096 -s 4.

Flink 作业管理器和应用程序管理器启动，但没有运行任务管理器.Flink Web 界面显示 0 表示任务管理器、任务槽和可用槽.但是，当我向 flink 集群提交作业时，任务管理器会被分配并运行，并且 Web UI 会按预期显示正确的值，并在作业完成后返回 0.

Flink Job Manager and Application Manager starts but there are no Task Managers running. The Flink Web interface shows 0 for task managers, task slots and slots available. However when I submit a job to flink cluster, then Task Managers get allocated and the job runs and the Web UI shows correct values as expected and goes back to 0 once the job is complete.

我希望任务管理器即使在没有提交作业的情况下也能运行，因为我想使用 Flink 的 REST API 根据可用槽值来监控和修改并行度.

I would like Task Managers to be running even when no Job is submitted as I want to use Flink's REST API to monitor and modify parallelism based on the available slots value.

是否有我遗漏的配置会阻止任务管理器一直运行?

Is there a configuration that I've missed which prevents Task Managers from running all the time?

推荐答案

Flink 的行为在 FLIP-6 中进行了更改，以允许动态资源分配.从 Flink 1.5.0 版本开始，可以将 mode: legacy 放在 flink-conf.yaml 中，以在旧模式下运行 Flink，其中任务管理器将与作业管理器一起启动.

Flink's behavior was changed in FLIP-6 to allow dynamic resource allocation. From Flink version 1.5.0, one can put mode: legacy in flink-conf.yaml to run Flink in the old mode where Task Managers will be started along with Job Manager.

这篇关于在 YARN 集群中提交作业之前，Flink TaskManager 不会启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！