为什么 kafka 中的 __consumer_offsets 主题没有传播到所有经纪人? [英] Why __consumer_offsets topic in kafka is not spreading to all the brokers?

问题描述

我有一个 3 zk 节点集群.以及 7 个 kafka 代理 节点.

I have a 3 zk nodes cluster. And 7 kafka broker nodes.

因此，当我创建任何主题时，我可以使用命令行参数设置副本因子和分区数.

So when I create any topic then I can set replica factor and no of partitions using command line arguements.

并且这些分区传播到所有 7 个代理.但是有一个主题，即 __consumer_offsets，它是自动创建的，并且它仅传播到 1 个代理(id=0)和 1 个复制.

And these partitions are spread to all the 7 brokers.But there is one topic i.e. __consumer_offsets,it is created automatically and it is spreading to only 1 broker(id=0) with 1 replication.

我知道我可以更改配置文件中复制因子的默认编号.

I am aware that I could change the default no of replication factor in config file.

但是，此主题仅复制到一个代理.那么可以更改哪些参数以将分区复制到所有代理.

But, this topic is replicated to only one broker. So what is the parameter which could be changed to replicate the partitions to all the brokers.

推荐答案

这对我来说真的很奇怪.我部署了 3 个 Kafka 代理集群，我的 __consumer_offsets 主题有 50 个分区(默认)，分布在 3 个代理上，并有 3 个(默认)作为副本.当您从单个 Kafka 代理开始时，您所描述的可能会发生，创建用于阅读主题的消费者，以便自动创建 __consumer_offsets 并且它将仅在代理 0(唯一的)上.之后，您添加新的 Kafka 代理:无需执行任何操作，__consumer_offsets 仍将保留在代理 0 上；如上所述，您需要使用 Kafka 手动重新分配分区.

It sounds really strange to me. I deployed a 3 Kafka brokers cluster and my __consumer_offsets topic has 50 partitions (the default) split across the 3 brokers and having 3 (the default) as replicas. What you are describing could happen when you start with a single Kafka broker, create consumers for reading topics so that the __consumer_offsets is automatically created and it will be just on the broker 0 (the only one). After that, you add new Kafka brokers: without doing anything the __consumer_offsets will still stay on broker 0; you need to use Kafka a manual partition reassignment for doing that as mentioned above.

这篇关于为什么 kafka 中的 __consumer_offsets 主题没有传播到所有经纪人?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！