什么是“重新平衡"?在Apache Kafka上下文中意味着什么? [英] What does "Rebalancing" mean in Apache Kafka context?

问题描述

我是Kafka的新用户，并且已经试用了2-3周.我相信目前我对Kafka的工作原理有一个很好的了解，但是在尝试为我自己的Kafka消费者使用该API之后(这虽然默默无闻，但我正在遵循新的KafkaConsumer的指导原则，适用于v 0.9(已在主干"存储库中列出)v)如果我有多个使用相同groupID的使用者，则我遇到了某个主题消耗的延迟问题.

I am a new user to Kafka and have been trialling it for about 2-3 weeks now. I believe at the moment I have a good understand of how Kafka works for the most part, but after attempting to fit the API for my own Kafka consumer (this is obscure but I'm following the guidelines for the new KafkaConsumer that is supposed to be available for v 0.9, which is out on the 'trunk' repo atm) I've had latency issues consuming from a topic if I have multiple consumers with the same groupID.

在此设置中，我的控制台始终记录有关重新平衡触发"的问题.当我将新的使用者添加到使用者组时是否发生重新平衡，是否触发了重新平衡以便弄清楚同一groupID中的哪个使用者实例将获得哪个分区，或者重新平衡完全用于其他用途?

In this setup, my console consistently logs issues regarding a 'rebalance triggering'. Do rebalances occur when I add new consumers to a consumer group and are they triggered in order to figure out which consumer instance in the same groupID will get which partitions or are rebalances used for something else entirely?

我也从 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design ，但我似乎无法理解，所以如果有人可以帮助我理解这一点，那将是非常感谢:

I also came across this passage from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design and I just can't seem to understand it, so if someone could help me make sense of it that would be much appreciated:

重新平衡是一组消费者实例的过程 (属于同一组)协调拥有互斥的 该组已订阅的主题分区集.在 消费群体成功进行重新平衡操作的结束，每个 所有订阅主题的分区将由一个消费者拥有 组中的实例.重新平衡的工作方式如下. 每个经纪人均被选为部分经纪人的协调人 消费群体.团体的统筹经纪人负责 用于协调对消费者组成员资格的重新平衡操作 订阅主题的更改或分区更改.也是 负责传达由此产生的分区所有权 进行平衡的组中所有消费者的配置 操作.

Rebalancing is the process where a group of consumer instances (belonging to the same group) co-ordinate to own a mutually exclusive set of partitions of topics that the group is subscribed to. At the end of a successful rebalance operation for a consumer group, every partition for all subscribed topics will be owned by a single consumer instance within the group. The way rebalancing works is as follows. Every broker is elected as the coordinator for a subset of the consumer groups. The co-ordinator broker for a group is responsible for orchestrating a rebalance operation on consumer group membership changes or partition changes for the subscribed topics. It is also responsible for communicating the resulting partition ownership configuration to all consumers of the group undergoing a rebalance operation.

推荐答案

当新使用者加入使用者组时，一组使用者将尝试重新平衡"负载，以将分区分配给每个使用者.如果在分配过程中使用方的数量发生变化，则重新平衡将失败并重试.此设置控制放弃前的最大尝试次数.

When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up.

用于此操作的命令是:rebalance.max.retries，默认情况下设置为4.

the command for this is: rebalance.max.retries and is set to 4 by default.

此外，如果满足以下条件，则可能会发生:

also, it might be happening if the following is true:

ZooKeeper会话超时.如果消费者在这段时间内没有对ZooKeeper进行心跳检查，则认为它已经死了，并且将会发生重新平衡.

ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

希望这会有所帮助！

这篇关于什么是“重新平衡"?在Apache Kafka上下文中意味着什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！