同一主题的2个kafka消费者的分区结构 [英] Partition structure for 2 kafka consumers on same topic

问题描述

如果我创建

• 2个kafka消费者实例
• 传递相同的属性
• 订阅同一主题

这两个Consumer实例(在diff组ID处)是否具有相似的分区结构，或者可以不同?

Will these 2 Consumer instances (at diff group Id), have similar partition structures, or could be different ?

即，如果我执行.assignment()，我在这两者上会得到相同的结果

i.e, if I do .assignment() will I get same result at both

我的实际问题陈述，我将在其中使用此验证

在我的应用程序中，我正在某个特定状态获得经纪人的抵销(这是通过我的第一个kafka消费者对象完成的).

In my application, I am attaining offset of broker, at a particular state(This is being done through my 1st kafka consumer object).

稍后，我将创建第二个kafka消费者对象，并使用它来遍历主题，从获得的更早偏移量中进行查找.

Later, I am creating the 2nd kafka consumer object, and using this to iterate over the topic, seeking from the earlier offset attained.

(因此，如果所提到的假设为假，我的逻辑将失败)

(So if the supposition mentioned at question is false, my logic will fail)

推荐答案

让我澄清一下.

Kafka有主题，消费者可以订阅它们.每个主题都有分区(您可以在创建分区时对其进行定义).当给定主题有多个分区时，每个主题分区都分配给使用者组中的使用者.如果您的使用者数量超过分区数量，则这些额外的使用者将变得空闲.

Kafka has topics, and the consumers can subscribe to them. Each topic has partitions (which you can define when you create them). When there are multiple partitions for a given topic, each topic partition is assigned to a consumer in a consumer group. If you have more consumers than the number of partitions, those extra consumers become idle.

如果希望两个Kafka使用者分别使用消息，则必须将它们添加到两个不同的使用者组中.如果使用者组中只有一个Kafka使用者，则所有分区都分配给该使用者.

If you want your two Kafka consumers to consume messages separately, you have to add them into two different consumer groups. If you have a single Kafka consumer within a Consumer group, all the partitions are assigned to that consumer.

因此，如果您希望两个消费者获得相同的结果，则可以将它们添加到两个不同的消费者组中.

So if you want to get the same result for the two consumers, you may add them into two different consumer groups.

这篇关于同一主题的2个kafka消费者的分区结构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！