什么是“再平衡"?在 Apache Kafka 上下文中是什么意思? [英] What does "Rebalancing" mean in Apache Kafka context?

问题描述

我是 Kafka 的新用户，现在已经试用了大约 2-3 周.我相信目前我对 Kafka 在很大程度上是如何工作的有一个很好的了解，但是在尝试为我自己的 Kafka 消费者安装 API 之后(这很模糊，但我正在遵循新的 KafkaConsumer 的指导方针，应该可用于 v 0.9，它在主干"repo atm 中)如果我有多个具有相同 groupID 的消费者，我会遇到从一个主题消费的延迟问题.

I am a new user to Kafka and have been trialling it for about 2-3 weeks now. I believe at the moment I have a good understand of how Kafka works for the most part, but after attempting to fit the API for my own Kafka consumer (this is obscure but I'm following the guidelines for the new KafkaConsumer that is supposed to be available for v 0.9, which is out on the 'trunk' repo atm) I've had latency issues consuming from a topic if I have multiple consumers with the same groupID.

在此设置中，我的控制台始终记录有关重新平衡触发"的问题.当我将新消费者添加到消费者组时是否会发生重新平衡，并且它们是否会被触发以找出同一 groupID 中的哪个消费者实例将获得哪些分区，或者重新平衡是否完全用于其他用途?

In this setup, my console consistently logs issues regarding a 'rebalance triggering'. Do rebalances occur when I add new consumers to a consumer group and are they triggered in order to figure out which consumer instance in the same groupID will get which partitions or are rebalances used for something else entirely?

我也从 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design 而我似乎无法理解它，所以如果有人可以帮助我理解它，那就是非常感谢:

I also came across this passage from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design and I just can't seem to understand it, so if someone could help me make sense of it that would be much appreciated:

重新平衡是一组消费者实例(属于同一组)协调拥有一个互斥的该组订阅的一组主题分区.在消费者组的成功重新平衡操作结束，每个所有订阅主题的分区将由单个消费者拥有组内的实例.重新平衡的工作方式如下.每个经纪人都被选为协调人的子集消费群体.一个小组的协调员经纪人负责用于编排对消费者组成员资格的重新平衡操作订阅主题的更改或分区更改.也是负责传达产生的分区所有权配置到正在重新平衡的组的所有消费者操作.

Rebalancing is the process where a group of consumer instances (belonging to the same group) co-ordinate to own a mutually exclusive set of partitions of topics that the group is subscribed to. At the end of a successful rebalance operation for a consumer group, every partition for all subscribed topics will be owned by a single consumer instance within the group. The way rebalancing works is as follows. Every broker is elected as the coordinator for a subset of the consumer groups. The co-ordinator broker for a group is responsible for orchestrating a rebalance operation on consumer group membership changes or partition changes for the subscribed topics. It is also responsible for communicating the resulting partition ownership configuration to all consumers of the group undergoing a rebalance operation.

推荐答案

当一个新的消费者加入一个消费者组时，这组消费者试图重新平衡"为每个消费者分配分区的负载.如果在此分配发生时消费者集发生变化，则重新平衡将失败并重试.此设置控制放弃前的最大尝试次数.

When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up.

这个命令是:rebalance.max.retries，默认设置为4.

the command for this is: rebalance.max.retries and is set to 4 by default.

此外，如果以下情况属实，则可能会发生这种情况:

also, it might be happening if the following is true:

ZooKeeper 会话超时.如果消费者在这段时间内未能向 ZooKeeper 发送心跳，则认为它已死亡并会发生重新平衡.

ZooKeeper session timeout. If the consumer fails to send a heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

希望这会有所帮助！

这篇关于什么是“再平衡"?在 Apache Kafka 上下文中是什么意思?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！