如何将主题从 kafka 集群复制到另一个 kafka 集群? [英] How to copy a topic from a kafka cluster to another kafka cluster?

问题描述

如 Kafka 文档所示，一种方法是通过 kafka.tools.MirrorMaker 来实现这一点.但是，我需要将一个主题(比如带有 1 个分区的 test)(其内容和元数据)从生产环境复制到没有连接的开发环境.不过，我可以在环境之间进行简单的文件传输.我的问题:如果我将 *.log 和 .index 从文件夹 test-0 移动到目标 Kafka 集群，是否足够好?或者还有更多我需要做的事情，比如我需要移动的元数据和 ZooKeeper 相关数据?

One way to do that as the Kafka documentation shows is through kafka.tools.MirrorMaker which can do that trick. However, I need to copy a topic (say test with 1 partition) (its content and meta data) from a production environment to a development environment where connectivity is not there. I could do simple file transfer between environments though. My question: if I move the *.log and .index from the folder test-0 to the destination Kafka cluster, is that good enough? Or there is more that I need to do like meta data and ZooKeeper-related data that I need to move too?

推荐答案

仅复制日志和索引是不够的 - kafka 在 zookeeper 中存储偏移量和主题元数据.MirrorMaker 实际上是一个非常简单的工具，它为源主题生成消费者以及为目标主题生成生产者，并运行直到所有消费者都消费了源队列.您找不到比迁移主题更简单的流程了.

Just copying the log and indexes will not suffice - kafka stores offsets and topic meta data in zookeeper. MirrorMaker is actually a quite simple tool, it spawns consumers to the source topic as well as producers to the target topic and runs until all consumers consumed the source queue. You can't find a simpler process to migrate a topic.

这篇关于如何将主题从 kafka 集群复制到另一个 kafka 集群?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！