消费者组如何在 kafka 中工作? [英] How consumer groups works in kafka?

问题描述

您好，我正在研究 kafka CLI，以清楚了解 kafka 的工作原理.我对消费者群体感到困惑.我创建了三个分区的主题.我将创建生产者以向主题提供一些数据.我第一次添加了一些数据，如下所示.

kafka-console-producer --broker-list 127.0.0.1:9092 --topic users>用户1kafka-console-producer --broker-list 127.0.0.1:9092 --topic 用户>用户2kafka-console-producer --broker-list 127.0.0.1:9092 --topic 用户>用户3

现在我的理解是 user1,user2,user3 会随机去三个不同的分区.

创建消费者组时，如下所示.

kafka-console-consumer --bootstrap-server localhost:9092 --topic users --group user_group

这会给我所有的用户 1、用户 2、用户 3.

现在在一个消费者组中，我可以拥有多个消费者.如果我在消费者组中有三个消费者，那么第一个消费者将从 partition1 中读取，第二个消费者将从消费者 2 中读取，然后第三个消费者将从消费者 3 中读取.这是我目前的理解.如果我的理解是正确的，那么演示上述行为的 cli 命令是什么?我只知道上面提到的一个命令会返回所有数据?如果我的上述理解是正确的，那么如果所有消费者都需要所有数据，那么如何获得它?有人可以帮助我理解这些概念.任何帮助将不胜感激.谢谢

解决方案

让我们首先了解分区与消费者的关系.

假设我有一个名为 T1 的主题，有 4 个分区和 1 个消费者组.在这种情况下，消费者组 1 将被分配从所有分区消费 -

现在，当我们将另一个消费者添加到同一个消费者组时，分区将在它们之间均匀分布 -

依此类推，当添加另一个消费者时，最多可达该主题中的分区数 -

在给定主题中添加超过分区数量的更多消费者将导致空闲消费者 -

这基本上意味着您受限于单个主题中的分区数量.

消费者如何加入消费者组?
当消费者想要加入消费者组时，他会发送一个JoinGroup向小组协调员提出要求.第一个加入组的人成为组长，他负责根据预定义的分配策略将分区的子集分配给每个消费者.
在决定了每个消费者的分区分配后，消费者领导会将分配分区列表发送给组协调器，他会将这些信息发送给组内的所有消费者.

如何选择分配策略?
Kafka 支持少数可以使用 partition.assignment.strategy 参数控制的分配策略.
这些政策是 RangeAssignor，RoundRobinAssignor 和 StickyAssignor 其中默认值为 RangeAssignor.

您可以在此有用的 博文.

如何查看?
我会推荐像 Kafka Manager 这样的工具，它可以帮助您将消费者与主题的关系可视化.>

Hi I am working on kafka CLI to get clear understanding how kafka works. I am confused on consumer groups. I have created topic with three partitions. I will create producer to feed some data to topic. First time I added some data as below.

kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user1
kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user2
kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user3

Now my understanding is user1,user2,user3 will go randomly to three different partitions.

When creating consumer group as below.

kafka-console-consumer --bootstrap-server localhost:9092 --topic users  --group user_group

This will give me all the user1,user2,user3.

Now inside one consumergroup I can have many consumers. If I have three consumer inside consumer group then first consumer will read from partition1, second consumer will read from consumer2 then third consumer will read from consumer3. This is my understanding so far. If my understanding is correct then what would be the cli command to demonstrate above behavior? I know only one command mentioned above which will return all the data? If my above understanding is correct then If all the consumer requires all the data then how to get it? Can someone help me to understand this concepts. Any help would be greatly appreciated. Thanks

解决方案

Let's start by understanding the Partitions to Consumers relationship.

Let's say I have a topic called T1 with 4 partitions, and 1 consumer group. In this case, Consumer Group 1 will by assigned to consume from all of the partitions -

Now, when we add another consumer to the same Consumer Group, the partitions will be evenly distributed between them -

And so on when adding another consumers, up to the number of partitions in that topic -

Adding more consumers beyond the amount of partitions in a given topic will cause an idle consumers -

That's basically means that you are bounded to the amount of partitions in a single topic.

How does consumers join Consumer Group?
When a consumer wants to join to a Consumer Group, he sends a JoinGroup request to the group coordinator. The first one to join the group becomes the group leader, and he is the one who is responsible for assigning a subset of the partitions to each consumer, based on a predefined assignment policy.
After deciding on the partition assignment for each consumer, the consumer leader will send the list of assignment partitions to the Group Coordinator and he will send this information to all the consumers within the group.

How to choose the assignment policy?
Kafka supports few assignment policy that can be controlled using the partition.assignment.strategy parameter.
The policies are RangeAssignor, RoundRobinAssignor and StickyAssignor where the default one is RangeAssignor.

You can read more about them on this useful blog post.

How to see it?
I would recommend tools like Kafka Manager that will help you visualize the consumer to topics relations.

这篇关于消费者组如何在 kafka 中工作?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！