消费者组成员没有分区 [英] Consumer group member has no partition

问题描述

我在同一个消费者组中启动了两个消费者，我订阅了20个主题(每个主题只有一个分区)

I launch two consumers on the same consumer group, i subscribe to 20 topics (each has only one partition)

仅在消费者上使用:

kafka-consumer-groups --bootstrap-server XXXXX:9092 --group foo --describe --members --verbose

kafka-consumer-groups --bootstrap-server XXXXX:9092 --group foo --describe --members --verbose

Note: This will not show information about old Zookeeper-based consumers.

CONSUMER-ID                                  HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT
rdkafka-07cbd673-6a16-4d55-9625-7f0925866540 /xxxxx rdkafka         20              arretsBus(0), capteurMeteo(0), capteurPointMesure(0), chantier(0), coworking(0), horodateur(
0), incident(0), livraison(0), meteo(0), metro(0), parkrelais(0), qair(0), rhdata(0), sensUnique(0), trafic(0), tramway(0), tweets(0), voieRapide(0), zone30(0), zoneRencontre(0)
rdkafka-9a543197-6c97-4213-bd59-cb5a48e4ec15 /xxxx    rdkafka         0 

我做错了什么?

推荐答案

好吧，我对这种行为做了一些阅读，知道为什么会发生很有趣.卡夫卡中有两种分区分配策略.

Ok, I did some reading around such behavior, and it's interesting to know why it happens. There are two kinds of partition assignment strategy in Kafka.

• 范围: Assigns to each consumer a consecutive subset of partitions from each topic it subscribes to. So if consumers C1 and C2 are subscribed to two topics, T1 and T2, and each of the topics has three partitions, then C1 will be assigned partitions 0 and 1 from topics T1 and T2, while C2 will be assigned partition 2 from those topics. Because each topic has an uneven number of partitions and the assignment is done for each topic independently, the first consumer ends up with more partitions than the second. This happens whenever Range assignment is used and the number of consumers does not divide the number of partitions in each topic neatly.

• Range: Assigns to each consumer a consecutive subset of partitions from each topic it subscribes to. So if consumers C1 and C2 are subscribed to two topics, T1 and T2, and each of the topics has three partitions, then C1 will be assigned partitions 0 and 1 from topics T1 and T2, while C2 will be assigned partition 2 from those topics. Because each topic has an uneven number of partitions and the assignment is done for each topic independently, the first consumer ends up with more partitions than the second. This happens whenever Range assignment is used and the number of consumers does not divide the number of partitions in each topic neatly.

RoundRobin: Takes all the partitions from all subscribed topics and assigns them to consumers sequentially, one by one. If C1 and C2 described previously used RoundRobin assignment, C1 would have partitions 0 and 2 from topic T1 and partition 1 from topic T2. C2 would have partition 1 from topic T1 and partitions 0 and 2 from topic T2. In general, if all consumers are subscribed to the same topics (a very common scenario), RoundRobin assignment will end up with all consumers having the same number of partitions (or at most 1 partition difference).

RoundRobin: Takes all the partitions from all subscribed topics and assigns them to consumers sequentially, one by one. If C1 and C2 described previously used RoundRobin assignment, C1 would have partitions 0 and 2 from topic T1 and partition 1 from topic T2. C2 would have partition 1 from topic T1 and partitions 0 and 2 from topic T2. In general, if all consumers are subscribed to the same topics (a very common scenario), RoundRobin assignment will end up with all consumers having the same number of partitions (or at most 1 partition difference).

默认策略是范围"，它说明了为什么看到这种分区分布.

The default strategy is Range, which explains why you are seeing such partition distribution.

所以，我做了一个小实验.我创建了两个控制台使用者，每个使用者都在听主题test1, test2, test3, test4，并且每个主题只有一个分区.如预期的那样，消费者1被分配了所有分区.

So, I did a small experiment. I created two console consumers each listening to topics test1, test2, test3, test4 and each topic has only one partition. As expected consumer-1 was assigned all partitions.

然后，我将分区策略更改为org.apache.kafka.clients.consumer.RoundRobinAssignor，并将其传递给控制台用户和voila，两个消费者现在每个都获得2个分区.

Then I changed the partitioning strategy to org.apache.kafka.clients.consumer.RoundRobinAssignor and passed it to both the console-consumers, and voila, both consumers now gets 2 partitions each.

更新: 糟糕，几分钟前还没有收到答复.

UPDATE: Oops didn't see it was already answered couple of minutes back.

这篇关于消费者组成员没有分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！