为什么kafka 0.8.2说每个分区都由一个消费者组中的一个消费者完全消费 [英] Why kafka 0.8.2 say that each partition is consumed by exactly one consumer in a consumer group

问题描述

在Apache Kafka 0.8.2 Office文档中，第5.6节分发，消费者和消费者群体" 小节，它表示

In Apache Kafka 0.8.2 office document, section 5.6 Distribution, Consumers and Consumer Groups subsection, it says that

一个组中的使用者将分区公平地划分为可能的是，每个分区仅由一个用户使用消费者群体.

The consumers in a group divide up the partitions as fairly as possible, each partition is consumed by exactly one consumer in a consumer group.

但是我发现实际上，消费者组中的多个消费者可以通过从同一主题分区发送FetchRequest来使用单个分区中的数据.

But I have found that in practice, it is possible that multiple consumers in a consumer group can consuming data from a single partition by sending FetchRequest from the same topic-partition.

以及随后的消费者ID注册中心小节

除了group_id之外，所有组中，每个消费者都会获得一个临时的，唯一的消费者ID(形式为hostname:uuid)以用于识别.消费者ID为在以下目录中注册.

In addition to the group_id which is shared by all consumers in a group, each consumer is given a transient, unique consumer_id (of the form hostname:uuid) for identification purposes. Consumer ids are registered in the following directory.

/consumers/[group_id]/ids/[consumer_id]-> {"topic1":#streams，...，"topicN":#streams}(临时节点)

/consumers/[group_id]/ids/[consumer_id] --> {"topic1": #streams, ..., "topicN": #streams} (ephemeral node)

它说每个消费者都有一个唯一的ID.但是，我在zookeeper中找不到这种结构.

It says there is a unique id for each consumer. However, I could not found such structure in zookeeper.

我不知道消费者何时开始注册?我使用的客户端库是kakfa-python 0.9.4.

I do not know when consumer start to register? The client library I used is kakfa-python 0.9.4.

推荐答案

可以提供帮助

(1)关于第二个问题. https://github.com/dpkp/kafka-python/issues/472 和问题38

(1) For your second question. https://github.com/dpkp/kafka-python/issues/472 And issue38

它说:协调中的消费者小组支持正在开发中."

It said "Coordinated Consumer Group support is under development."

(2)第一个问题.

它说:这是通过将主题中的分区分配给消费者组中的消费者来实现的，以便每个分区被该组中的一个消费者完全消费."(声明A).这取决于客户端工具.在某些kafka客户中，这可能不正确.我只是有python和cpp的经验.如果实施了分组，则每条消息将由该分组中的一个消费者使用.在一组中的使用者之间分配分区的方法不同.当分区的数量超过使用者数量时，语句A可能是正确的.但是，也有可能在新分区加入或离开现有组时重新分配分区.在这种情况下，分区A可以先由消费者A消费，然后再由消费者B消费，这是可能的.在某些客户端中，您可以选择分配算法，例如循环调度等.

It said "This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. "(statement A). This depends on clients implements. This may be not right in some kafka clients. I just have experience in python and cpp. If group was implemented, each message is consumed by exactly one consumer in the group. How to assign partitions between consumers in one group is different. When there are more partitions than consumers, Statement A may be right. But it is also possible that the partitions may be re-assigned when new partitions join or leave the existing group. In this case, partition A may be consumed by consumer A firstly and then consumed by consumer B, which is possible. In some clients, you can choose the assignment algorithms, such as round-robin, and so on.

这篇关于为什么kafka 0.8.2说每个分区都由一个消费者组中的一个消费者完全消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！