在YARN群集中提交作业之前，Flink TaskManager不会启动 [英] Flink TaskManagers do not start until job is submitted in YARN cluster

问题描述

我正在使用Amazon EMR在YARN上运行Flink Cluster.我的设置包括用于1个主节点和2个核心节点的m4.large实例.我已经使用以下命令在YARN上启动了Flink CLuster.

I am using Amazon EMR to run Flink Cluster on YARN. My setup consists of m4.large instances for 1 master and 2 core nodes. I have started the Flink CLuster on YARN with the command: flink-yarn-session -n 2 -d -tm 4096 -s 4.

Flink作业管理器和应用程序管理器启动，但是没有任务管理器在运行. Flink Web界面为任务管理器，任务插槽和可用插槽显示0.但是，当我将作业提交给flink群集时，任务管理器将得到分配，作业将运行，Web UI将按预期显示正确的值，并在作业完成后返回0.

Flink Job Manager and Application Manager starts but there are no Task Managers running. The Flink Web interface shows 0 for task managers, task slots and slots available. However when I submit a job to flink cluster, then Task Managers get allocated and the job runs and the Web UI shows correct values as expected and goes back to 0 once the job is complete.

我希望即使没有提交作业也可以运行任务管理器，因为我想使用Flink的REST API根据可用的插槽值监视和修改并行性.

I would like Task Managers to be running even when no Job is submitted as I want to use Flink's REST API to monitor and modify parallelism based on the available slots value.

是否缺少我要阻止任务管理器一直运行的配置?

Is there a configuration that I've missed which prevents Task Managers from running all the time?

推荐答案

在FLIP-6中更改了Flink的行为，以允许动态分配资源.从Flink 1.5.0版开始，可以将mode: legacy放在flink-conf.yaml中，以在旧模式下运行Flink，在旧模式下，任务管理器将与作业管理器一起启动.

Flink's behavior was changed in FLIP-6 to allow dynamic resource allocation. From Flink version 1.5.0, one can put mode: legacy in flink-conf.yaml to run Flink in the old mode where Task Managers will be started along with Job Manager.

这篇关于在YARN群集中提交作业之前，Flink TaskManager不会启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！