Kafka 消费者默认组 ID [英] Kafka Consumer default Group Id

问题描述

我正在使用 Apache Kafka 及其 Java 客户端，我看到消息在属于同一组的不同 Kafka 使用者之间进行负载平衡(即共享相同的组 ID).

在我的应用程序中，我需要所有消费者阅读所有消息.

所以我有几个问题:

• 如果我没有在 Consumer Properties 中设置任何 group id，那么 Kafka Consumer 会被赋予什么 group id?

• 有单一的默认值吗?

• 客户端是否每次都创建一个随机值?

• 我是否需要为每个消费者创建不同的 ID 以确保每个消费者都能收到所有消息?

感谢您的回答.

你是对的:如果没有设置消费者组ID，Kafka应该抱怨.

但是，我发现如果组ID为空，Java客户端将其设置为空字符串"以避免出现问题.很明显，这是我正在寻找的默认值.

令我所有的消费者感到惊讶，即使我没有设置他们的 groupIds(所以他们都是 groupId == "")似乎收到了生产者写的所有消息.

我还是无法解释:有什么建议吗?

解决方案

如果我在 Consumer Properties 中没有设置任何 group id，那么 Kafka Consumer 会被赋予什么 group id?>

kafka 消费者不会有任何消费者组.相反，您会收到此错误:配置的 groupId 无效

有单一的默认值吗?

是的，可以参考kafka的consumer.properties文件.默认的消费者组id是:group.id=test-consumer-group

客户端是否每次都创建一个随机值?

不，对于启动 Kafka 0.9.0.x 消费者的 Java 客户端，groupId 似乎是必需的.你可以参考这个JIRA:https://issues.apache.org/jira/browse/KAFKA-2648

我是否需要为每个消费者创建不同的 ID 以确保每个消费者都能收到所有消息?

是的，如果所有消费者使用相同的组 ID，主题中的消息将在这些消费者之间分发.换句话说，每个消费者都将获得一个不重叠的消息子集.在同一组中拥有更多的消费者会增加并行度和消费的整体吞吐量.另一方面，如果每个消费者都在自己的组中，则每个消费者都将获得所有消息的完整副本.

I'm working with Apache Kafka and its Java client and I see that messages are load balanced across different Kafka Consumers belonging to the same group (i.e. sharing the same group id).

In my application I need all consumers to read all messages.

So I have several questions:

• if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

• Is there a single default value?

• Does the client create a random value each time?

• Do I need to create a different id for each consumer to be sure that each one receives all messages?

EDIT: Thank you for your answers.

You are correct: if one doesn't set the consumer group id, Kafka should complain.

However, I have found out that if the group id is null, the Java client sets it to the empty string "" to avoid problems. So apparently that is the default value I was looking for.

Surprising all my consumers, even if I don't set their groupIds (and so they are all with groupId == "") seem to receive all the messages the producer writes.

I still can't explain this: any suggestions?

解决方案

if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

The kafka consumer will not have any consumer group. Instead you will get this error : The configured groupId is invalid

Is there a single default value?

Yes, you can see the consumer.properties file of kafka for reference. The default consumer group id is: group.id=test-consumer-group

Does the client create a random value each time?

No, groupId seems to be mandatory for Java client starting Kafka 0.9.0.x consumers. You can refer to this JIRA: https://issues.apache.org/jira/browse/KAFKA-2648

Do I need to create a different id for each consumer to be sure that each one receives all messages?

Yes, if all consumers use the same group id, messages in a topic are distributed among those consumers. In other words, each consumer will get a non-overlapping subset of the messages. Having more consumers in the same group increases the degree of parallelism and the overall throughput of consumption. On the other hand, if each consumer is in its own group, each consumer will get a full copy of all messages.

这篇关于Kafka 消费者默认组 ID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！