如何将主题从kafka群集复制到另一个kafka群集? [英] How to copy a topic from a kafka cluster to another kafka cluster?

问题描述

Kafka文档显示的一种实现方法是通过 kafka.tools.MirrorMaker 来完成.但是，我需要将一个主题(例如具有1个分区的 test )(其内容和元数据)从生产环境复制到不存在连通性的开发环境中.我可以在环境之间进行简单的文件传输.我的问题:如果将 *.log 和 .index 从文件夹 test-0 移至目标Kafka集群，是否足够好?还是还有其他需要做的事情，例如元数据和与ZooKeeper相关的数据?

One way to do that as the Kafka documentation shows is through kafka.tools.MirrorMaker which can do that trick. However, I need to copy a topic (say test with 1 partition) (its content and meta data) from a production environment to a development environment where connectivity is not there. I could do simple file transfer between environments though. My question: if I move the *.log and .index from the folder test-0 to the destination Kafka cluster, is that good enough? Or there is more that I need to do like meta data and ZooKeeper-related data that I need to move too?

推荐答案

仅复制日志和索引是不够的-kafka将偏移量和主题元数据存储在zookeeper中.MirrorMaker实际上是一个非常简单的工具，它不仅使使用者产生源主题，还使产生者产生目标主题，并运行直到所有使用者消耗了源队列.您找不到更简单的迁移主题的过程.

Just copying the log and indexes will not suffice - kafka stores offsets and topic meta data in zookeeper. MirrorMaker is actually a quite simple tool, it spawns consumers to the source topic as well as producers to the target topic and runs until all consumers consumed the source queue. You can't find a simpler process to migrate a topic.

这篇关于如何将主题从kafka群集复制到另一个kafka群集?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！