Kafka添加新使用者不会重新平衡负载 [英] Kafka adding new consumers does not rebalance the load

问题描述

我有一个主题，其中包含2个经纪人上的3个分区.(卡夫卡版本:0.8.1)

I have a topic with 3 partitions on 2 brokers. (Kafka version: 0.8.1)

使用不同的用户guid(例如FC42B34DD7658503E040970A2C437358)作为分区键批量添加消息.(约1万条消息)

Messages are bulk added using different user guid (like this: FC42B34DD7658503E040970A2C437358) as partition key. (about 10k messages)

在加载消息时，我有一个运行中的使用者(consumer1)，它开始很好地处理消息.
然后，我启动了另一个具有相同消费者组ID的消费者(consumer2).

While loading the messages, I have one running consumer (consumer1), it starts handling the messages fine.
Then I started another consumer (consumer2) with same consumer group id.

我注意到的是，Consumer1停止处理消息，而Consumer2开始处理所有消息.

What I noticed is that, consumer1 stops handling the messages, and consumer2 starts handling all the messages.

当我停止consumer2时，consumer1接管并恢复消息处理.

When I stop consumer2, then consumer1 took over and resumes message processing.

我期望两个消费者都应该分配负载.

I was expecting both consumers should distribute the load.

任何线索可能出在哪里?谢谢.

Any clue where could be the problem? Thanks.

推荐答案

我不确定在没有对Zookeeper群集进行一些检查的情况下，您的消费者中到底发生了什么，但是我怀疑一种可能的情况，即您的生产者可能无法平均分配消息到分区.

I am not sure what happens in your consumers exactly without some inspections on your Zookeeper cluster, but I can suspect one possible scenario that your producers may not evenly distribute messages to the partitions.

一个分区由一个消费者组中的一个消费者拥有.所有者使用者称为 partition owner ，所有进入分区的消息均由其分区所有者专门使用.(有关详细信息，请参阅Kafka 0.8.1文档中的消费者.

A partition is owned by a single consumer in a single consumer group. The owner consumer is called a partition owner, and all messages come in to a partition is exclusively consumed by its partition owner. (For more information, refer to Consumers in Kafka 0.8.1 documentation.

我们说有三个分区A，B，C和两个使用者1和2，生产者仅向分区B发送消息.

Let us say there are three partitions A, B, C and two consumers 1 and 2, and the producers only send messages to the partition B.

只有消费者1时，分区1的所有消息都被消费者1消耗.

When there is only consumer 1, all messages of the partition B are consumed by the consumer 1.

在介绍使用者2时，现在通过使用者重新平衡算法将分区B分配给了使用者2.由于您的生产者仅将消息发送到分区B，因此，消费者2成为唯一使用消息的消费者.

When you introduce consumer 2, now the partition B is assigned to the consumer 2 by a consumer rebalancing algorithm. Since your producers send messages only to the partition B, now the consumer 2 becomes the only consumer that consumes messages.

在停止使用方2之后，分区B再次分配给使用方1，并且所有消息都被使用方1使用.

After you stop the consumer 2, the partition B is assigned to the consumer 1 again, and all messages are consumed by the consumer 1.

以上情况是我能想到的.检查您的生产者实现是否存在分发问题.

The above scenario is what I can think of. Check whether your producer implementation has a distribution problem.

这篇关于Kafka添加新使用者不会重新平衡负载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！