将具有自定义配置的kafka主题镜像为独立应用程序 [英] Mirror kafka topics with custom configuration as a standalone application

问题描述

我想将一些话题从一个经纪人反映到另一个经纪人.只是所有主题的一部分.有一个用于此的 MirrorMaker 工具.但我也想更改目标主题名称.另外，自定义消息处理程序已经做到了.但是，它不符合我的需求.

I want to mirror some topics from one broker to another. Just part of all topics. There's an existed MirrorMaker tool for this. But I also want to change destination topic names. Also, Custom message handler already does it. Nevertheless, it doesn't fit my needs.

对我有几个要求:

1. 可以提供每个目标主题替代的镜像主题
2. 动态检测新的源主题
3. 运行一个独立的应用程序(例如Java，Spring Boot)，而不是 CLI方法

至少有人对其中任何一个有想法吗?

Does anybody have ideas at least for one of them?

推荐答案

据我所知，没有现成的工具可以完成您提到的所有事情.Confluent的Replicator可能具有所有这些功能，但是您必须支付企业许可证的费用.此处有更多详细信息:

As far as I know, there is no existing tool that does all the things you mention. Confluent's Replicator probably has all that functionality, but you'd have to pay for an Enterprise license. More details here:

https://www.confluent.io/connector/confluent-kafka-复制器/

如果您不想支付Confluent许可证的费用，我的建议是构建自己的应用程序，该应用程序可以使用和生成Kafka信息.您将拥有完全的控制权，可以用Java编写它，可以实现所需的任何逻辑(自动检测新主题，在目标主题上动态更改名称，等等)，还可以将其部署为普通JVM应用程序最适合您的方式(普通JVM，AWS EC2，Docker/Kubernetes应用程序等)

My recommendation, if you don't want to pay for the Confluent license, is to build your own application that both consumes and produces info Kafka. You'd have full control, you can write it in Java, you can implement whatever logic you want (detect new topics automatically, change names dynamically on the destination topics, etc), and you can deploy it as a normal JVM application in whatever way suits you most (normal JVM, AWS EC2, Docker/Kubernetes app, etc)

这篇关于将具有自定义配置的kafka主题镜像为独立应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！