Kafka 消息迁移 [英] Kafka Message migration

问题描述

我们目前在 Apache Kafka 0.10.1.1 上运行.我们正在迁移到 Confluent Platform 5.X.新集群完全设置在不同的物理节点集上.

We are currently operating on Apache Kafka 0.10.1.1. We are migrating to Confluent Platform 5.X. The New cluster is setup completely on different set of physical nodes.

虽然我们已经在努力升级 API，但我们的应用程序使用 spring-boot，我们正试图弄清楚我们如何迁移消息?我需要在目标集群中保持相同的消息顺序.

While we are already working on upgrading the API(s), our application uses spring-boot, we are trying to figure out how do we migrate the messages? I need to maintain the same ordering of messages in the Target Cluster.

1. 我可以简单地复制消息吗?
2. 是否需要将消息重新发布到 Target 集群才能成功保留?
3. 还能做什么?

推荐答案

假设新集群中的topic定义完全一样(即:nbr of partitions、retention等..)和Producer对消息的hash函数key 会将您的消息传递到同一个分区(如果您有空键，这将是一个无赖，因为它最终会出现在一个随机分区中)，您可以简单地从旧的 kafka 代理主题中最早使用并生成到您的新主题使用自定义消费者/生产者或诸如 logstash 之类的工具的新集群.

Assuming the topic definition in the new cluster is exactly the same (i.e: nbr of partitions, retention, etc..) and the Producer hashing function on the message key will deliver your message to the same partition (will be a bummer if you have null keys because it'll end up in a random partition), you can simply consume from earliest from your old kafka broker topic and produce to your new topic in the new cluster using a custom consumer/producer or some tool like logstash.

如果您想更加确定获得相同的排序，您应该每个主题只使用一个使用者，并且如果您的使用者支持单线程运行，甚至更好(可能避免竞争条件).

If you want to be extra sure to get the same ordering, you should only use only one consumer per topic and if your consumer supports single threaded run, even better (might avoid racing conditions).

您也可以尝试更常见的解决方案，例如 MirrorMaker，但请注意，MirrorMaker 订购保证金额为:

You might also try more common solutions like MirrorMaker but be advised that MirrorMaker ordering guarantees amount to:

然而，MirrorMaker 进程将保留并使用消息键进行分区，以便保留顺序基于每个键.

The MirrorMaker process will, however, retain and use the message key for partitioning so order is preserved on a per-key basis.

注意:如第一个解决方案中所述以及 cricket_007 所说，它仅在您使用默认分区程序并希望在新集群中继续使用它时才有效.

Note: As stated in the first solution and as cricket_007 said, it will only work if you were using the default partitioner and wish to keep using it in the new cluster.

最后，如果一切顺利，您可以从旧的 kafka 代理手动复制您的消费者偏移量，并在您的新集群消费者组中定义它们.

In the end, if everything goes OK, you can manually copy your consumer offsets from the old kafka broker and define them on your new cluster consumer groups.

免责声明:这纯粹是理论上的.我从未尝试过具有这种严格要求的迁移.

Disclaimer: This is purely theoritical. I've never tried a migration with this sort of hard requirements.

这篇关于Kafka 消息迁移的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！