多个Kafka消费者可以从分区读取相同的消息吗 [英] Can multiple Kafka consumers read same message from the partition

问题描述

我们计划编写一个 Kafka 消费者(java)，它读取 Kafka 队列以执行消息中的操作.

We are planning to write a Kafka consumer(java) which reads Kafka queue to perform an action which is in the message.

由于消费者独立运行，消息是否一次只由一个消费者处理?否则所有消费者处理相同的消息，因为他们在分区中有自己的偏移量.

As the consumers run independently, will the message is processed by only one consumer at a time? Else all the consumers process the same message as they have their own offset in the partition.

请帮我理解.

推荐答案

取决于 组 ID.假设您有一个包含 12 个分区的主题.如果您有 2 个具有相同 Group Id 的 Kafka 消费者，他们都将读取 6 个分区，这意味着他们将读取不同的分区集 = 不同的消息集.如果您有 4 个具有相同 Group Id 的 Kafka 用户，他们每个人都将读取三个不同的分区等.

It depends on Group ID. Suppose you have a topic with 12 partitions. If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set of partitions = different set of messages. If you have 4 Kafka cosnumers with the same Group Id, each of them will all read three different partitions etc.

但是当您设置不同的组 ID 时，情况就会改变.如果您有两个具有不同 Group Id 的 Kafka 消费者，他们将读取所有 12 个分区，而彼此之间不会产生任何干扰.这意味着两个消费者将独立读取完全相同的消息集.如果您有四个具有不同组 ID 的 Kafka 消费者，他们都将读取所有分区等.

But when you set different Group Id, the situation changes. If you have two Kafka consumers with different Group Id they will read all 12 partitions without any interference between each other. Meaning both consumers will read the exact same set of messages independently. If you have four Kafka consumers with different Group Id they will all read all partitions etc.

这篇关于多个Kafka消费者可以从分区读取相同的消息吗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！