卡夫卡重新分配__consumer_offsets不正确? [英] Kafka reassignment of __consumer_offsets incorrect?

问题描述

我对__consumer_offsets主题的kafka-reassignment-paritions如何工作感到困惑?

I am confused with how kafka-reassignment-paritions works for __consumer_offsets topic?

我从1个zk和1个kafka代理开始，创建一个复制= 1，分区= 1的测试主题.消费和生产.效果很好.

I start with 1 zk and 1 kafka broker, create a test topic with replication=1, partition=1. consume and produce. works fine.

我看到创建了__consumer_offsets主题.

I see __consumer_offsets topic created.

现在，我添加一个具有 offsets.topic.replication.factor = 2 的代理.我运行

Now I add a second broker with, offsets.topic.replication.factor=2. I run the,

kafka-reassign-partitions --zookeeper zookeeper1:2181 --topics-to-move-json-file topic-to-move.json --broker-list"101,102" --generate

生成的重新分配看起来不正确.即使有2个实时代理，也仅显示一个副本.

The generated reassignment does not look right. Only shows one replica even though there are 2 live brokers.

我希望为每个分区获取以下副本:[101，102]或[201，101]

I was hoping to get following replicas for each partition: [101, 102] or [201, 101]

{
  "version": 1,
  "partitions": [
    {
      "topic": "__consumer_offsets",
      "partition": 19,
      "replicas": [101]
    },
    {
      "topic": "__consumer_offsets",
      "partition": 30,
      "replicas": [102]
    },
    {
      "topic": "__consumer_offsets",
      "partition": 47,
      "replicas": [101]
    }, ...

赞赏任何建议.

-Vms

推荐答案

如果要增加主题的复制因子，请执行以下步骤:

If you want to increase the replication factor for a topic, follow steps below:

1. 创建一个包含重新分配计划的json文件.就您而言，该文件可能如下所示:

1. Create a json file containing the reassignment plan. In your case, the file might look like:

{"version":1, "partitions":[
  {"topic":"__consumer_offsets","partition":0,"replicas":[101,102]}, 
  {"topic":"__consumer_offsets","partition":1,"replicas":[102,101]},
  {"topic":"__consumer_offsets","partition":2,"replicas":[101,102]},
  {"topic":"__consumer_offsets","partition":3,"replicas":[102,101]},
  ...
  {"topic":"__consumer_offsets","partition":49,"replicas":[101,102]}
]}

在下面运行命令以增加此内部主题的RF:

Run command below to increase RF for this internal topic:

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassign.json --execute

然后运行 kafka-topics.sh --describe 来查看复制因子是否达到2.

Then run kafka-topics.sh --describe to see if replication factor is bumped up to 2.

这篇关于卡夫卡重新分配__consumer_offsets不正确?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！