Kafka Broker 找不到集群 ID 并在 docker 重启后创建新的集群 ID [英] Kafka Broker doesn't find cluster id and creates new one after docker restart

问题描述

我已经使用 kafka 代理和动物园管理员创建了 docker，以使用运行脚本启动它.如果我重新启动，它会正常启动并运行正常(Windows -> WSL -> 两个 tmux 窗口，一个会话).如果我关闭 kafka 或 zookeeper 并重新启动它会正常连接.

I've created docker with kafka broker and zookeeper to start it with run script. If I do fresh start it starts normally and runs ok (Windows -> WSL -> two tmux windows, one session). If I shut down kafka or zookeeper and start it again it will connect normally.

当我停止 docker 容器时出现问题(docker stop my_kafka_container).然后我从我的脚本 ./run_docker 开始.在开始之前的那个脚本中，我删除旧容器 docker rm my_kafka_container 然后 docker run.

Problem occurs when I stop docker container (docker stop my_kafka_container). Then I start with my script ./run_docker. In that script before start I delete old container docker rm my_kafka_containerand then docker run.

Zookeeper 正常启动，并且在文件 meta.properties 中，它具有上次启动时的旧集群 ID，但 kafka 代理由于某种原因无法通过 znode cluster/id 找到此 ID 并创建新的，这不是存储在元数据中的.特性.我得到

Zookeeper starts normally and in file meta.properties it has old cluster id from previous start up, but kafka broker for some reason cannot find by znode cluster/id this id and creates new one which is not that which is stored in meta.properties. And I get

  ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID m1Ze6AjGRwqarkcxJscgyQ doesn't match stored clusterId Some(1TGYcbFuRXa4Lqojs4B9Hw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:84)
        at kafka.Kafka.main(Kafka.scala)
[2020-01-04 15:58:43,303] INFO shutting down (kafka.server.KafkaServer)

如何避免broker修改集群id?

How to avoid broker change it cluster id?

推荐答案

我在使用 Docker 时遇到了同样的问题.此问题自 Kafka 2.4 以来出现，因为添加了检查以查看 Zookeeper 中的集群 ID 是否匹配.它将集群 ID 存储在 meta.properties 中.

I had the same issue when using Docker. This issue occurs since Kafka 2.4 because a check was added to see if the Cluster ID in Zookeeper matches. It's storing the cluster id in meta.properties.

这可以通过使 Zookeeper 数据持久化而不仅仅是 Zookeeper 日志来解决.例如.使用以下配置:

This can be fixed by making the Zookeeper data persistent and not only the Zookeeper logs. E.g. with the following config:

volumes:
  - ~/kafka/data/zookeeper_data:/var/lib/zookeeper/data
  - ~/kafka/data/zookeeper_log:/var/lib/zookeeper/log

您还应该删除 Kafka 日志中的 meta.properties 文件一次，以便 Kafka 从 Zookeeper 检索正确的集群 ID.之后，ID 应该匹配，您就不必再执行此操作了.

You should also remove the meta.properties file in the Kafka logs once so that Kafka retrieves the right cluster id from Zookeeper. After that the IDs should match and you don't have to do this anymore.

您也可能会遇到 2.4 中也添加的 snapshot.trust.empty 错误.您可以通过添加 snapshot.trust.empty=true 设置或在升级到 2.4 之前使 Zookeeper 数据持久化来解决此问题.

You may also run into a snapshot.trust.empty error which was also added in 2.4. You can solve this by either adding the snapshot.trust.empty=true setting or by making the Zookeeper data persistent before doing the upgrade to 2.4.

这篇关于Kafka Broker 找不到集群 ID 并在 docker 重启后创建新的集群 ID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！