Kafka消费者在各个主题之间保持平衡 [英] Kafka Consumers are balanced across topics

问题描述

我正在创建6个具有相同组ID的消费者.所有消费者都在订阅2个主题，每个主题有3个分区.由于有2个主题有6个使用者和6个分区，因此我希望所有使用者都可以使用.但是我看不到所有的消费者都在使用，有没有办法强迫我重新平衡呢?

I am creating 6 consumers with same group-id. All the consumers are subscribing to 2 topics with 3 partitions each. Since there are 6 consumers and 6 partitions across 2 topics I am expecting all the consumers to be used. But I don't see all the consumers being used, is there a way I can force it to re balance ?

我正在使用kafka 0.10.2.0

I am using kafka 0.10.2.0

推荐答案

未跨主题在同一消费者组中为消费者分配分区.跟随你发生的事情... 您有t1_p0，t1_p1和t1_p2(在topic1上的分区)和t2_p0，t2_p1和t2_p2(在topic2上的分区).然后，您有一个包含6个使用者c1，c2，... c6的使用者组. 当他们订阅topic1时，可能会发生c1，c2和c3获得t1_p0，t1_p1和t1_p2的情况.其他3个使用者(c4，c5，c6)由于没有足够的分区而处于空闲状态. 当相同的使用者订阅topic2时，可能会发生相同的情况:c1，c2和c3获得t1_p0，t1_p1和t1_p2；其他3个使用者(c4，c5，c6)由于没有足够的分区而处于空闲状态. 我知道您想让c4，c5和c6从主题2获取这些分区，但是正如我所说的，分区分配不会跨主题发生，而是针对同一主题中的分区. 在这种情况下，您有c4，c5和c6处于空闲状态并且未被使用.

Assigning partitions to consumers in the same consumer group is not done across topics. Following what is happening to you ... You have t1_p0, t1_p1 and t1_p2 (partitions on topic1) and t2_p0, t2_p1 and t2_p2 (partitions on topic2). Then you have a consumer group with 6 consumers c1, c2, ... c6. When they subscribe to topic1, it can happen that c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. When the same consumers subscribe to topic2, it can happen the same: c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. I know that you would like having c4, c5 and c6 getting these partitions from topic 2 but as I said partitions assignment doesn't happen across topics but for partitions in the same topic. In this scenario you have c4, c5 and c6 in idle and not used.

这篇关于Kafka消费者在各个主题之间保持平衡的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！