为什么 kafka 0.8.2 说每个分区只被一个消费者组中的一个消费者消费 [英] Why kafka 0.8.2 say that each partition is consumed by exactly one consumer in a consumer group

问题描述

在 Apache Kafka 0.8.2 办公文档中，第 5.6 节分发，Consumers and Consumer Groups 小节，它说

In Apache Kafka 0.8.2 office document, section 5.6 Distribution, Consumers and Consumer Groups subsection, it says that

一组中的消费者公平地划分分区可能，每个分区都由一个消费者在一个消费群体.

The consumers in a group divide up the partitions as fairly as possible, each partition is consumed by exactly one consumer in a consumer group.

但我发现，在实践中，一个消费者组中的多个消费者有可能通过从同一主题分区发送 FetchRequest 来消费来自单个分区的数据.

But I have found that in practice, it is possible that multiple consumers in a consumer group can consuming data from a single partition by sending FetchRequest from the same topic-partition.

在随后的Consumer Id Registry小节

除了 group_id 是所有消费者共享的组，每个消费者都被赋予一个短暂的、唯一的消费者 ID(表单主机名:uuid)用于识别目的.消费者 ID 是在以下目录中注册.

In addition to the group_id which is shared by all consumers in a group, each consumer is given a transient, unique consumer_id (of the form hostname:uuid) for identification purposes. Consumer ids are registered in the following directory.

/consumers/[group_id]/ids/[consumer_id] --> {"topic1": #streams, ...,"topicN": #streams}(临时节点)

/consumers/[group_id]/ids/[consumer_id] --> {"topic1": #streams, ..., "topicN": #streams} (ephemeral node)

它表示每个消费者都有一个唯一的 ID.但是，我在zookeeper中找不到这样的结构.

It says there is a unique id for each consumer. However, I could not found such structure in zookeeper.

不知道消费者什么时候开始注册?我使用的客户端库是 kakfa-python 0.9.4.

I do not know when consumer start to register? The client library I used is kakfa-python 0.9.4.

推荐答案

可能会有所帮助

(1) 对于你的第二个问题.https://github.com/dpkp/kafka-python/issues/472和问题38

(1) For your second question. https://github.com/dpkp/kafka-python/issues/472 And issue38

它说正在开发协调消费者组的支持."

It said "Coordinated Consumer Group support is under development."

(2) 对于你的第一个问题.

(2) For your first question.

它说这是通过将主题中的分区分配给消费者组中的消费者来实现的，以便每个分区都被组中的一个消费者消费."(语句A).这取决于客户端实现.这可能不适用于某些 kafka 客户端.我只有 python 和 cpp 的经验.如果实施了群组，则每条消息 都由群组中的一个消费者消费.如何在一组消费者之间分配分区是不同的.当分区数多于消费者时，语句 A 可能是正确的.但也有可能在新分区加入或离开现有组时重新分配分区.在这种情况下，分区 A 可能会先被消费者 A 消费，然后被消费者 B 消费，这是可能的.在某些客户端中，您可以选择分配算法，例如轮询等.

It said "This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. "(statement A). This depends on clients implements. This may be not right in some kafka clients. I just have experience in python and cpp. If group was implemented, each message is consumed by exactly one consumer in the group. How to assign partitions between consumers in one group is different. When there are more partitions than consumers, Statement A may be right. But it is also possible that the partitions may be re-assigned when new partitions join or leave the existing group. In this case, partition A may be consumed by consumer A firstly and then consumed by consumer B, which is possible. In some clients, you can choose the assignment algorithms, such as round-robin, and so on.

这篇关于为什么 kafka 0.8.2 说每个分区只被一个消费者组中的一个消费者消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！