Kafka消息迁移 [英] Kafka Message migration
问题描述
我们当前正在使用Apache Kafka 0.10.1.1. 我们正在迁移到Confluent Platform5.X. 新群集完全在不同的物理节点集上设置.
We are currently operating on Apache Kafka 0.10.1.1. We are migrating to Confluent Platform 5.X. The New cluster is setup completely on different set of physical nodes.
虽然我们已经在进行API升级，但我们的应用程序使用的是spring-boot，但我们正在尝试弄清楚如何迁移消息?我需要在目标群集中保持相同的消息顺序.
While we are already working on upgrading the API(s), our application uses spring-boot, we are trying to figure out how do we migrate the messages? I need to maintain the same ordering of messages in the Target Cluster.
- 我可以简单地复制邮件吗?
- 是否需要将消息重新发布到目标集群才能成功保留?
- 还有什么可以做的?
推荐答案
假设新集群中的主题定义完全相同(即:分区，保留等的nbr个.)和消息中的Producer哈希函数key会将您的消息传递到同一个分区(如果您使用null键，则消息将不胜枚举，因为它将最终出现在一个随机分区中)，您可以简单地从最早的kafka broker主题中最早消费，并在中产生新的主题使用自定义使用者/生产者或诸如logstash之类的工具来创建新集群.
Assuming the topic definition in the new cluster is exactly the same (i.e: nbr of partitions, retention, etc..) and the Producer hashing function on the message key will deliver your message to the same partition (will be a bummer if you have null keys because it'll end up in a random partition), you can simply consume from earliest from your old kafka broker topic and produce to your new topic in the new cluster using a custom consumer/producer or some tool like logstash.
如果您想确保自己获得相同的订购，则每个主题只能使用一个消费者，并且如果您的消费者支持单线程运行，甚至更好(可能避免出现竞争状况).
If you want to be extra sure to get the same ordering, you should only use only one consumer per topic and if your consumer supports single threaded run, even better (might avoid racing conditions).
您也可以尝试使用更常见的解决方案，例如MirrorMaker，但建议MirrorMaker的订购保证为:
You might also try more common solutions like MirrorMaker but be advised that MirrorMaker ordering guarantees amount to:
但是，MirrorMaker进程将 保留并使用消息密钥进行分区，以便保留顺序 在每个键的基础上.
The MirrorMaker process will, however, retain and use the message key for partitioning so order is preserved on a per-key basis.
注意:如第一个解决方案中所述，并且如cricket_007所述，只有在使用默认分区程序并希望在新群集中继续使用它的情况下，此方法才有效.
Note: As stated in the first solution and as cricket_007 said, it will only work if you were using the default partitioner and wish to keep using it in the new cluster.
最后，如果一切顺利，您可以从旧的kafka代理手动复制消费者补偿，并在新的集群消费者组中定义它们.
In the end, if everything goes OK, you can manually copy your consumer offsets from the old kafka broker and define them on your new cluster consumer groups.
免责声明:这纯粹是理论上的.我从来没有尝试过具有这种严格要求的迁移.
Disclaimer: This is purely theoritical. I've never tried a migration with this sort of hard requirements.
这篇关于Kafka消息迁移的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
