在群集节点上进行Flink作业分配 [英] Flink job distribution over cluster nodes
问题描述
我们有4个作业正在3个节点上运行，每个节点有4个插槽，
We have 4 jobs that are running over 3 nodes with 4 slots per each,
在Flink 1.3.2上，作业按节点平均分配. 升级到flink 1.5之后，每个作业都在单个节点上运行(如果没有插槽，则转移到另一个节点上)
On Flink 1.3.2 the jobs were evenly distributed per node. After upgrading to flink 1.5 , each job is running on a single node (with a carry over to another if there are no slots left)
有没有办法恢复到平均分配? 作业负载不均，导致某些节点比其他节点更努力地工作.
Is there a way to return to an even distribution? The jobs are not evenly by load which cause some nodes to work harder than other.
推荐答案
An answer I received from flink mailing list Re: Flink 1.5 job distribution over cluster nodes
Shachar
Hi Shachar,
使用Flink 1.5，我们增加了资源弹性.这意味着Flink是 现在可以在集群管理框架上分配新容器 例如Yarn或Mesos.由于这些更改(这也适用于 独立模式)，Flink不再需要一组固定的 TaskManagers，因为如果需要，它将启动新容器(不会 在独立模式下工作).因此，系统很难 对扩展属于单个作业的插槽做出任何决定 跨多个TM.当您考虑到这一点时，甚至会变得更加困难 像您这样的工作可能会受益于这种策略，而 其他用户则可以从其插槽的共同位置中受益.它变得更多 如果您想将wrt调度到多个作业，则很复杂 该系统不完全了解，因为它们是 按顺序提交.因此，Flink当前假定该插槽 任何TaskManager都可以满足请求.
with Flink 1.5 we added resource elasticity. This means that Flink is now able to allocate new containers on a cluster management framework like Yarn or Mesos. Due to these changes (which also apply to the standalone mode), Flink no longer reasons about a fixed set of TaskManagers because if needed it will start new containers (does not work in standalone mode). Therefore, it is hard for the system to make any decisions about spreading slots belonging to a single job out across multiple TMs. It gets even harder when you consider that some jobs like yours might benefit from such a strategy whereas others would benefit from co-locating its slots. It gets even more complicated if you want to do scheduling wrt to multiple jobs which the system does not have full knowledge about because they are submitted sequentially. Therefore, Flink currently assumes that slots requests can be fulfilled by any TaskManager.
干杯，直到
这篇关于在群集节点上进行Flink作业分配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
