Kafka 对 __consumer_offsets 的重新分配不正确? [英] Kafka reassignment of __consumer_offsets incorrect?
问题描述
我对 __consumer_offsets 主题的 kafka-reassignment-paritions 如何工作感到困惑?
I am confused with how kafka-reassignment-paritions works for __consumer_offsets topic?
我从 1 个 zk 和 1 个 kafka broker 开始,创建一个具有 replication=1、partition=1 的测试主题.消费和生产.工作正常.
I start with 1 zk and 1 kafka broker, create a test topic with replication=1, partition=1. consume and produce. works fine.
我看到创建了 __consumer_offsets 主题.
I see __consumer_offsets topic created.
现在我添加第二个代理,offsets.topic.replication.factor=2.我运行,
Now I add a second broker with, offsets.topic.replication.factor=2. I run the,
kafka-reassign-partitions --zookeeper zookeeper1:2181 --topics-to-move-json-file topic-to-move.json --broker-list "101,102" --generate
生成的重新分配看起来不正确.即使有 2 个实时代理,也只显示一个副本.
The generated reassignment does not look right. Only shows one replica even though there are 2 live brokers.
我希望为每个分区获得以下副本:[101, 102] 或 [201, 101]
I was hoping to get following replicas for each partition: [101, 102] or [201, 101]
{
"version": 1,
"partitions": [
{
"topic": "__consumer_offsets",
"partition": 19,
"replicas": [101]
},
{
"topic": "__consumer_offsets",
"partition": 30,
"replicas": [102]
},
{
"topic": "__consumer_offsets",
"partition": 47,
"replicas": [101]
}, ...
感谢任何建议.
-Vms
推荐答案
如果要增加主题的复制因子,请按照以下步骤操作:
If you want to increase the replication factor for a topic, follow steps below:
创建一个包含重新分配计划的 json 文件.在您的情况下,该文件可能如下所示:
Create a json file containing the reassignment plan. In your case, the file might look like:
{"version":1, "partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[101,102]},
{"topic":"__consumer_offsets","partition":1,"replicas":[102,101]},
{"topic":"__consumer_offsets","partition":2,"replicas":[101,102]},
{"topic":"__consumer_offsets","partition":3,"replicas":[102,101]},
...
{"topic":"__consumer_offsets","partition":49,"replicas":[101,102]}
]}
运行以下命令以增加此内部主题的 RF:
Run command below to increase RF for this internal topic:
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassign.json --execute
然后运行 kafka-topics.sh --describe
以查看复制因子是否提高到 2.
Then run kafka-topics.sh --describe
to see if replication factor is bumped up to 2.
这篇关于Kafka 对 __consumer_offsets 的重新分配不正确?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!