重新启动后，Kafka主题不再存在 [英] Kafka topic no longer exists after restart

问题描述

我在具有3个服务器/代理的本地kafka集群中创建了一个主题 通过在我的kafka安装目录中运行以下命令

I created a topic in my local kafka cluster with 3 servers / brokers by running the following from my kafka installation directory

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic user-activity-tracking-pipeline

一切正常，因为我能够从自己的主题中产生和传达消费者信息.重新启动机器后，我通过在终端中运行以下命令从kafka安装目录中启动了捆绑的zookeeper

Everything worked fine as I was able to produce and consumer messages from my topic. After restarting my machine, I started bundled zookeeper from kafka installation directory by running the following in the terminal

bin/zookeeper-server-start.sh config/zookeeper.properties

通过在终端的kafka安装目录中执行以下命令，启动了属于该集群的3台服务器

Started 3 servers belonging to the cluster by executing the following in terminal from kafka installation directory

env JMX_PORT=10001 bin/kafka-server-start.sh config/server1.properties
env JMX_PORT=10002 bin/kafka-server-start.sh config/server2.properties
env JMX_PORT=10003 bin/kafka-server-start.sh config/server3.properties

现在，当我通过从kafka安装目录在终端中运行以下命令来列出可用主题时，

Now, when I list available topics by running the following in terminal from kafka installation directory,

bin/kafka-topics.sh --zookeeper localhost:2181 --list

结果为空！

这是相关的服务器1配置条目.服务器2和服务器3的值非常相似

Here are the relevant server 1 configuration entries. The values for server 2 and server 3 are quite similar

broker.id=1
listeners=PLAINTEXT://:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs-broker-1
num.partitions=2
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

重启后我确实注意到日志文件在下面，所以没有清理任何东西

I do notice log files under after restart so nothing was cleaned up

/tmp/kafka-logs-broker-1
/tmp/kafka-logs-broker-2
/tmp/kafka-logs-broker-3

我想知道为什么以前创建的主题 用户活动跟踪管道" 不再存在?

I am wondering why the previously created topic "user-activity-tracking-pipeline" doesn't exist any more when I try to list it?

推荐答案

kafka-topics.sh实际上是在后台使用Zookeeper数据来回答查询.理由是，单个经纪人通常本身无法获得足够的信息来完整地描述主题.

kafka-topics.sh actually uses zookeeper data under the hood to answer the query. The rationale being that a single broker generally can't have enough information by itself to describe topics completely.

如果在重新启动过程中丢失了(我怀疑是因为提到了一个新的Zookeeper启动而丢失的)Zookeeper数据，那么kafka-topics现在完全是盲目的，看不到以前的kafka数据.

If you lost (which I suspect you did, since you mention a new zookeeper start) zookeeper data during your restart process, kafka-topics is now totally blind and can't see former kafka data.

检查发生情况的最好方法是在查询时实际执行kafka的操作！启动您的zookeeper客户端(就像执行./zkCli.sh一样简单，然后键入ls /brokers/topics.如果为空，则ZK数据将丢失.

The best way to check what's happening is to actually do what kafka is doing when you query it ! Launch your zookeeper client (it's as simple as doing ./zkCli.sh, and type ls /brokers/topics. If it's empty, your ZK data is lost.

这篇关于重新启动后，Kafka主题不再存在的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！