某个代理宕机后如何更改主题领导者或删除分区? [英] How change topic leader or remove partition after some broker down?

问题描述

我们有 kafka 集群，其中包含 4 个代理和一些具有副本因子 1 和 10 分区的主题.在某一时刻，我们的 4 个服务器中有 2 个带有 kafka 集群 - 失败.所以现在我们有 2 个具有相同主题的经纪人.当 im 运行命令时./kafka_topics.sh --zookeeper 本地主机:2181 --describe我我明白了:

We have kafka cluster with 4 brokers and some topics with replica factor 1 and 10 partitions. At one moment 2 of 4 our servers with kafka cluster - fail. So now we have 2 brokers with same topics. When im run command ./kafka_topics.sh --zookeeper localhost:2181 --describe im get this:

Topic:outcoming-notification-error-topic        PartitionCount:10       ReplicationFactor:1     Configs:
    Topic: outcoming-error-topic       Partition: 0    Leader: 2       Replicas: 2     Isr: 2
    Topic: outcoming-error-topic       Partition: 1    Leader: 3       Replicas: 3     Isr: 3
    Topic: outcoming-error-topic       Partition: 2    Leader: 4       Replicas: 4     Isr: 4
    Topic: outcoming-error-topic       Partition: 3    Leader: 1       Replicas: 1     Isr: 1
    Topic: outcoming-error-topic       Partition: 4    Leader: 2       Replicas: 2     Isr: 2
    Topic: outcoming-error-topic       Partition: 5    Leader: 3       Replicas: 3     Isr: 3
    Topic: outcoming-error-topic       Partition: 6    Leader: 4       Replicas: 4     Isr: 4
    Topic: outcoming-error-topic       Partition: 7    Leader: 1       Replicas: 1     Isr: 1
    Topic: outcoming-error-topic       Partition: 8    Leader: 2       Replicas: 2     Isr: 2
    Topic: outcoming-error-topic       Partition: 9    Leader: 3       Replicas: 3     Isr: 3

如何删除Leader 2...4?或者我可能需要为这个领导者删除分区，但是如何?

How can i delete Leader 2...4 ? or may be i need delete partition for this Leader , but how ?

更新...

我们还使用 kafka_exporter 来监控 kafka 与 prometheus.在 kafka_exporter 的日志中有 2 个经纪人关闭后，我们收到此错误:

Also we use kafka_exporter for monitoring kafka with prometheus. After 2 brokers down in log of kafka_exporter we get this errors:

level=error msg="Cannot get oldest offset of topic outcoming-error-topic partition  10: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:296"

推荐答案

你可以使用 Kafka 的 kafka-reassign-partitions.sh 来做到这一点.您有两种方法，一种是生成新任务的建议，另一种是 手动指定特定分区的领导者.

You could use Kafka's kafka-reassign-partitions.sh to do that. You have two ways, one is generating a proposal of new assignments, and the other one is manually specifying the leaders for specific partitions.

第一种方法，如 kafka 文档 中所指定，遵循以下逻辑:

The first method, as specified on the kafka docs, follows this logic:

1.1 生成建议的分区重新分配配置

首先，您应该创建一个 json 文件，例如链接中提供的.我们将其命名为 topics.json.

First, you should create a json file such as the provided in the link. Let's name it topics.json.

{
  "topics": [{"topic": "foo1"},
            {"topic": "foo2"}],
  "version":1
}

这将告诉 kafka 您愿意从哪些主题重新分配其分区.在示例中，他希望 Kafka 为主题 foo1 和 foo2 提出建议.

This will tell kafka what are the topics you are willing to rellocate their partitions from. In the example, he wants Kafka to make a proposal for topics foo1 and foo2.

使用该 json，调用该工具并在命令中设置活动代理列表:

With that json, call the tool and set the active broker list in the command:

kafka-reassign-partitions.sh --zookeeper $ZK_HOSTS 
--topics-to-move-json-file topics.json --broker-list "1,2,3,4,5" --generate

这将输出 Kafka 的建议，您可以将其保存到另一个 .json 文件中.例如:

This will output Kafka's proposal, which you can save into another .json file. For example:

{
  "version":1,
  "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
              {"topic":"foo1","partition":0,"replicas":[5,6]},
              {"topic":"foo2","partition":2,"replicas":[5,6]},
              {"topic":"foo2","partition":0,"replicas":[5,6]},
              {"topic":"foo1","partition":1,"replicas":[5,6]},
              {"topic":"foo2","partition":1,"replicas":[5,6]}]
}

如果您愿意(或认为这是正确的做法，因为该工具并不完美)，您可以手动修改某些分配.将json保存到一个文件中，例如reassign-example.json，下一步会用到.

You can manually modify some of the assignments, if you want to (or think it's the proper think to do, as the tool is not perfect). Save the json into a file, for example, reassign-example.json, which will be used in the next step.

1.2.执行提议的分区重新分配

让Kafka执行提案并移动分区.为此，执行:

Let's make Kafka execute the proposal and move the partitions. For that, execute:

bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOSTS 
 --reassignment-json-file reassign-example.json --execute

这将执行在 reassign-example.json 文件中定义的分区移动.

This will execute the partition movement defined on the reassign-example.json file.

第二种方法相当简单，但您必须手动确定要重新分配的分区.例如，如果您希望主题 XXX 的分区 1 移动到代理 5 和 6，您可以创建一个 json 文件(manual-reassign.json)，例如:

The second method is fairly easier, but you must manually identify the partitions you want to reassign. For example, if you want partition 1 of topic XXX to move to brokers 5 and 6, you could create a json file (manual-reassign.json) such as:

{"version":1,"partitions":[{"topic":"XXX","partition":1,"replicas":[5,6]}]}

它的启动方式与之前的方法相同:

The way it's launched is the same as in the previous method:

bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOSTS 
 --reassignment-json-file manual-reassign.json --execute

这篇关于某个代理宕机后如何更改主题领导者或删除分区?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！