消费者重新平衡在卡夫卡如何运作? [英] How does consumer rebalancing work in Kafka?

问题描述

添加或关闭新的消费者/经纪人后，Kafka会触发重新平衡操作. Kafka正在重新平衡阻止操作吗?重新平衡操作进行期间，Kafka消费者是否受阻?

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress?

推荐答案

取决于您所说的受阻".如果您的意思是触发重新平衡时现有连接已关闭"，那么答案是肯定的.不幸的是，当前的Kafka的重新平衡算法并不完善.

Depends on what you mean by "blocked". If you mean "are existing connections closed when rebalance is triggered" then the answer is yes. The current Kafka's rebalancing algorithm is unfortunately imperfect.

这是消费者重新平衡过程中发生的事情.

Here is what is happening during consumer rebalance.

假设我们有一个包含10个分区(0-9)的主题，并且有一个使用它的使用者(将其命名为consumer1).当第二个使用者出现(consumer2)时，重新平衡任务将为他们两个触发(consumer1获取一个事件，consumer2进行初始重新平衡).现在，consumer1关闭所有现有连接(甚至那些将很快重新打开的连接)，并释放所有10个分区在Zookeeper中的分区所有权.

Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) consuming it. When a second consumer appears (consumer2) the rebalance task triggers for both of them (consumer1 gets an event, consumer2 does the initial rebalance). Now consumer1 closes all the existing connections (even those that will be reopened soon) and releases the partition ownership in Zookeeper for all 10 partitions.

然后，它运行分区分配算法，并决定应声明哪些分区，然后再次在Zookeeper中声明该分区的所有权.如果声明成功，则consumer1开始获取他的新分区.

Then it runs the partition assignment algorithm and decides what partitions should be claimed and claims the partition ownership in Zookeeper again. If the claim was successful consumer1 starts fetching his new partitions.

同时consumer2也运行分区分配算法，并尝试在Zookeeper中声明其分区.仅当consumer1释放这些分区上的所有权时，声明才会成功.声明成功后，consumer2开始提取，或者在给定的重试次数内未能声明分区时，您将获得rebalance failed after n retries异常.

Meanwhile consumer2 runs the partition assignment algorithm as well and tries to claim his partitions in Zookeeper as well. Claim will succeed only when consumer1 releases the ownership on these partitions. When the claim is successful consumer2 starts fetching, or if it fails to claim partitions within a given amount of retries you get a rebalance failed after n retries exception.

不仅仅是关闭连接并释放分区的所有权而不再拥有consumer1，它不必要地关闭了他的所有连接并仅使用较少数量的分区重新启动.添加分区的情况相同(当我们使用通配符过滤器使用时，出现新主题)-关闭所有连接，然后重新打开，而不仅仅是打开新连接.

As you noticed instead of just closing connections and releasing ownership for partitions consumer1 does not own anymore, it unnecessarily closes ALL his connections and restarts with just a lower amount of partitions. The same story with adding partitions (when we consume by a wildcard filter and new topic appears) - ALL connections are closed and then opened again instead of just opening new ones.

所以我希望这能回答您的问题-当重新平衡开始时，获取会停止.

So I hope this answers your question - fetching stops when rebalance kicks in.

这篇关于消费者重新平衡在卡夫卡如何运作?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！