在任务管理器之间均匀分布 Flink 操作符 [英] Distribute a Flink operator evenly across taskmanagers

问题描述

我正在一个由 15 台机器组成的裸机集群上对 Flink 流应用程序进行原型设计.我正在使用具有 90 个任务槽 (15x6) 的纱线模式.

I'm prototyping a Flink streaming application on a bare-metal cluster of 15 machines. I'm using yarn-mode with 90 task slots (15x6).

该应用从单个 Kafka 主题读取数据.Kafka 主题有 15 个分区，因此我将源操作符的并行度也设置为 15.但是，我发现 Flink 在某些情况下会将消费者任务的 2-4 个实例分配给同一个任务管理器.这会导致某些节点受网络限制(Kafka 主题正在提供大量数据，而机器只有 1G 网卡)和整个数据流的瓶颈.

The app reads data from a single Kafka topic. The Kafka topic has 15 partitions, so I set the parallelism of the source operator to 15 as well. However, I found that Flink in some cases assigns 2-4 instances of the consumer task to the same taskmanager. This causes certain nodes to become network-bound (the Kafka topic is serving high volume of data and the machines only have 1G NICs) and bottlenecks in the entire data flow.

有没有办法强制"或以其他方式指示 Flink 在所有任务管理器之间平均分配任务，也许是循环?如果没有，有没有办法手动将任务分配到特定的任务管理器插槽?

Is there a way to "force" or otherwise instruct Flink to distribute a task evenly across all taskmanagers, perhaps round robin? And if not, is there a way to manually assign tasks to specific taskmanager slots?

推荐答案

Flink 不允许手动分配任务槽，因为在处理失败的情况下，它可以将任务分配给剩余的任务管理器.

Flink does not allow manually assign task slots as in case of failure handling, it can distribute the task to remaining task managers.

但是，您可以通过在 flink-conf.yaml 中设置 cluster.evenly-spread-out-slots: true 来均匀分配工作负载.这适用于 Flink >= 1.9.2.

However, you can distribute the workload evenly by setting cluster.evenly-spread-out-slots: true in flink-conf.yaml. This works for Flink >= 1.9.2.

要使其工作，您可能还需要设置:
taskmanager.numberOfTaskSlots 等于每台机器的可用 CPU 数量，并且
parallelism.default 等于集群中的 CPU 总数.

To make it work, you may also have to set:
taskmanager.numberOfTaskSlots equal to the number of available CPUs per machine, and
parallelism.default equal to the the total number of CPUs in the cluster.

这篇关于在任务管理器之间均匀分布 Flink 操作符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！