Flink 1.3在YARN上运行单个作业如何设置每个TaskManager的任务插槽数 [英] Flink 1.3 running a single job on YARN how to set the number of Task Slots per TaskManager
问题描述
I can set the number of yarn nodes / task managers with the above parameter -yn
. However I want to know whether it is possible to set the number of task slots per task manager. When I use the parallelsim (-p
) parameter it only sets the overall parallelism. And the number of task slots is computed by dividing this value by the number of provided task managers. I tried using the dynamic properties (-yD
) parameter which is supposed to "allow the user to specify additional configuration values" like this:
-yD -Dtaskmanager.numberOfTaskSlots=8
但这不会覆盖flink-conf.yaml
中给出的值.
在flink上运行单个任务时,是否有任何方法可以指定每个TaskManager的任务插槽数(而不是更改配置文件)?
还提供了使用-yD
参数哪些动态属性有效的文档?
But this does not overwrite the value given in the flink-conf.yaml
.
Is there any way to specify the number of task slots per TaskManager when running a single on flink (other than changing the config file)?
Also is there a documentation which dynamic properties are valid using the -yD
parameter?
推荐答案
You can use the settings of yarn-session, here, prefixed by y
to submit Flink job on YARN cluster. For example the command,
flink run -m yarn-cluster -yn 5 -yjm 768 -ytm 1400 -ys 2 -yqu streamQ my_program.jar
将提交具有5
容器,768m
内存用于 jobmanager 的my_program.jar
Flink应用程序,1400m
内存和2
cpu内核用于 taskmanagers ,并将使用预定义的YARN队列streamQ
上的nodemanagers的资源.请参阅我对此帖子的回答重要信息.
will submit my_program.jar
Flink application with 5
containers, 768m
memory for the jobmanager, 1400m
memory and 2
cpu core for taskmanagers, each and will use the resources of nodemanagers on predefined YARN queue streamQ
. See my answer to this post for other important information.
这篇关于Flink 1.3在YARN上运行单个作业如何设置每个TaskManager的任务插槽数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!