Kafka 中的消费者再平衡是如何工作的? [英] How does consumer rebalancing work in Kafka?

问题描述

当一个新的消费者/代理被添加或关闭时，Kafka 会触发重新平衡操作.Kafka Rebalancing 是阻塞操作吗?在进行重新平衡操作时，Kafka 消费者是否被阻止?

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress?

推荐答案

取决于您所说的阻止"是什么意思.如果您的意思是触发重新平衡时现有连接是否已关闭"，那么答案是肯定的.不幸的是，目前Kafka的重新平衡算法并不完善.

Depends on what you mean by "blocked". If you mean "are existing connections closed when rebalance is triggered" then the answer is yes. The current Kafka's rebalancing algorithm is unfortunately imperfect.

以下是消费者再平衡期间发生的事情.

Here is what is happening during consumer rebalance.

假设我们有一个包含 10 个分区 (0-9) 的主题，并且有一个消费者(我们将其命名为 consumer1)消费它.当第二个消费者出现 (consumer2) 时，重新平衡任务会为他们两个触发(consumer1 获得一个事件，consumer2 进行初始重新平衡).现在 consumer1 关闭所有现有连接(甚至那些即将重新打开的连接)并释放 Zookeeper 中所有 10 个分区的分区所有权.

Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) consuming it. When a second consumer appears (consumer2) the rebalance task triggers for both of them (consumer1 gets an event, consumer2 does the initial rebalance). Now consumer1 closes all the existing connections (even those that will be reopened soon) and releases the partition ownership in Zookeeper for all 10 partitions.

然后它运行分区分配算法并决定应该声明哪些分区并再次声明在 Zookeeper 中的分区所有权.如果声明成功，consumer1 开始获取他的新分区.

Then it runs the partition assignment algorithm and decides what partitions should be claimed and claims the partition ownership in Zookeeper again. If the claim was successful consumer1 starts fetching his new partitions.

与此同时，consumer2 也运行分区分配算法，并尝试在 Zookeeper 中声明他的分区.只有当 consumer1 释放这些分区的所有权时，声明才会成功.当声明成功时 consumer2 开始获取，或者如果它未能在给定的重试次数内声明分区，您将收到 rebalance failed after n retries 异常.

Meanwhile consumer2 runs the partition assignment algorithm as well and tries to claim his partitions in Zookeeper as well. Claim will succeed only when consumer1 releases the ownership on these partitions. When the claim is successful consumer2 starts fetching, or if it fails to claim partitions within a given amount of retries you get a rebalance failed after n retries exception.

正如您所注意到的，它不仅关闭连接并释放 consumer1 不再拥有的分区的所有权，而是不必要地关闭了他的所有连接，并以较少的分区重新启动.添加分区的相同故事(当我们通过通配符过滤器使用并出现新主题时) - 所有连接都关闭然后再次打开，而不是只打开新连接.

As you noticed instead of just closing connections and releasing ownership for partitions consumer1 does not own anymore, it unnecessarily closes ALL his connections and restarts with just a lower amount of partitions. The same story with adding partitions (when we consume by a wildcard filter and new topic appears) - ALL connections are closed and then opened again instead of just opening new ones.

所以我希望这能回答您的问题 - 当重新平衡开始时获取停止.

So I hope this answers your question - fetching stops when rebalance kicks in.

这篇关于Kafka 中的消费者再平衡是如何工作的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！