创建 Kafka 主题导致没有领导者 [英] Creating a Kafka topic results in no leader

问题描述

我使用的是 Kafka v0.9.0.1 (Scala v2.11) 和 com.101tec:zkclient v0.7.我正在尝试使用 AdminUtils 创建一个 kafka 主题.我的代码如下.

String zkServers = "node1:2181,node2:2181,node3:2181,node4:2181";整数 sessionTimeout = (int)TimeUnit.SECONDS.toMillis(10L);整数 connectionTimeout = (int)TimeUnit.SECONDS.toMillis(8L);ZkSerializer zkSerializer = ZKStringSerializer$.MODULE$;布尔值 isSecureKafkaCluster = false;字符串主题=测试"；整数分区 = 1;整数复制 = 3;ZkClient zkClient = new ZkClient(zkServers, sessionTimeout, connectionTimeout, zkSerializer);ZkUtils zkUtils = new ZkUtils(zkClient, new ZkConnection(zkServers), isSecureKafkaCluster)if(!AdminUtils.topicExists(zkUtils, topic)) {AdminUtils.createTopic(zkUtils, topic, partitions, replications, new Properties());}

主题实际上是通过以下命令验证创建的.

bin/kafka-topics.sh --describe --zookeeper node1:2181 --topic 测试

然而，输出并不如预期.

<前>主题:测试 PartitionCount:1 ReplicationFactor:1 配置:主题:测试分区:0 领导者:-1 副本:4 Isr:

如果我使用脚本.

bin/kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic topic1

然后我看到以下内容.

Topic:test1 PartitionCount:1 ReplicationFactor:3 配置:主题:test1 分区:0 领导者:2 副本:2,3,4 Isr:2

对我做错了什么有什么想法吗?结果是，如果我使用 Producer 将 ProducerRecord 发送到主题，则该主题上不会显示任何内容.

解决方案

我遇到了同样的问题.
解决方案:

1. 清理 zk 元信息 (/brokers/topic)

2. 清理所有/data 目录以删除属于该主题的所有主题分区文件夹

3. 立即重启整个 kafka 集群的所有代理.

4. 重新创建该主题.

这解决了我的问题.而且我认为根本原因是 kafka 本身的缺陷无法处理干净的删除主题(自 v1.0.0 以来已修复).

即使使用 Kafka(>= v1.0.0)，如果您正在删除一个空主题或者您的 kafka 集群处于极端负载下，有时删除主题也会卡住.
解决方案就像重新启动控制器代理一样简单.(您始终可以通过 get/controller 找到 ZK 下的控制器代理:/controller).所以只需重启一个代理而不是整个 kafka 集群.

I am using Kafka v0.9.0.1 (Scala v2.11) and the com.101tec:zkclient v0.7. I am trying to use AdminUtils to create a kafka topic. My code is the following.

String zkServers = "node1:2181,node2:2181,node3:2181,node4:2181";
Integer sessionTimeout = (int)TimeUnit.SECONDS.toMillis(10L);
Integer connectionTimeout = (int)TimeUnit.SECONDS.toMillis(8L);
ZkSerializer zkSerializer = ZKStringSerializer$.MODULE$;
Boolean isSecureKafkaCluster = false;
String topic = "test";
Integer partitions = 1;
Integer replication = 3;

ZkClient zkClient = new ZkClient(zkServers, sessionTimeout, connectionTimeout, zkSerializer);
ZkUtils zkUtils = new ZkUtils(zkClient, new ZkConnection(zkServers), isSecureKafkaCluster)
if(!AdminUtils.topicExists(zkUtils, topic)) {
 AdminUtils.createTopic(zkUtils, topic, partitions, replications, new Properties());
}

The topic is actually created as verified by the following command.

bin/kafka-topics.sh --describe --zookeeper node1:2181 --topic test

However, the output is not as expected.

Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test Partition: 0    Leader: -1  Replicas: 4 Isr: 

If I use the script.

bin/kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic topic1

Then I see the following.

Topic:test1 PartitionCount:1    ReplicationFactor:3 Configs:
Topic: test1    Partition: 0    Leader: 2   Replicas: 2,3,4 Isr: 2

Any ideas on what I'm doing wrong? The effect is that if I use a Producer to send a ProducerRecord to the topic, nothing shows up on the topic.

解决方案

I had the same issue.
Solution:

1. Clean zk meta info (/brokers/topic)

2. Clean all /data dir to remove all topic-partition folders belongs to that topic

3. Restart the whole kafka cluster all brokers at once.

4. Recreate that topic.

This solved my problem. And I think the root cause was the defect from kafka itself failing to handle clean removal topics (this has been fixed since v1.0.0).

Edit: even with Kafka(>= v1.0.0), sometimes deleting topic will stuck if you are deleting an empty topic or if your kafka cluster is under extreme load.
solution would be as simple as restarting the controller broker. (you can always find the controller broker under ZK: /controller by get /controller). so just restarting one broker instead of the whole kafka cluster.

这篇关于创建 Kafka 主题导致没有领导者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！