同一主题的 2 个 kafka 消费者的分区结构 [英] Partition structure for 2 kafka consumers on same topic

问题描述

如果我创建

• 2 个 kafka 消费者实例
• 传递相同的属性
• 订阅同一主题

这 2 个消费者实例(在 diff group Id 中)是否具有相似的分区结构，或者可能不同?

Will these 2 Consumer instances (at diff group Id), have similar partition structures, or could be different ?

即，如果我执行 .assignment() 我会得到相同的结果

i.e, if I do .assignment() will I get same result at both

我的实际问题陈述，我将在其中使用此验证

在我的应用程序中，我在特定状态下获得了代理的偏移量(这是通过我的第一个 kafka 消费者对象完成的).

In my application, I am attaining offset of broker, at a particular state(This is being done through my 1st kafka consumer object).

稍后，我创建了第二个 kafka 消费者对象，并使用它来迭代主题，从先前获得的偏移量中寻找.

Later, I am creating the 2nd kafka consumer object, and using this to iterate over the topic, seeking from the earlier offset attained.

(所以如果问题中提到的假设是错误的，我的逻辑就会失败)

(So if the supposition mentioned at question is false, my logic will fail)

推荐答案

让我来澄清一下.

Kafka 有主题，消费者可以订阅它们.每个主题都有分区(您可以在创建它们时定义).当给定主题有多个分区时，每个主题分区都分配给消费者组中的一个消费者.如果您的消费者多于分区数量，那么这些额外的消费者就会闲置.

Kafka has topics, and the consumers can subscribe to them. Each topic has partitions (which you can define when you create them). When there are multiple partitions for a given topic, each topic partition is assigned to a consumer in a consumer group. If you have more consumers than the number of partitions, those extra consumers become idle.

如果你想让你的两个 Kafka 消费者分别消费消息，你必须将它们添加到两个不同的消费者组中.如果消费者组中有一个 Kafka 消费者，则所有分区都分配给该消费者.

If you want your two Kafka consumers to consume messages separately, you have to add them into two different consumer groups. If you have a single Kafka consumer within a Consumer group, all the partitions are assigned to that consumer.

所以如果你想得到两个消费者相同的结果，你可以将他们加入到两个不同的消费者组中.

So if you want to get the same result for the two consumers, you may add them into two different consumer groups.

这篇关于同一主题的 2 个 kafka 消费者的分区结构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！