使用 mqtt 协议和 kafka 作为消息代理 [英] using mqtt protocol with kafka as a message broker

问题描述

我们如何使用 mqtt 协议和 kafka 作为消息代理?

客户端(android/ios/desktop java 应用程序等)将使用 mqtt phao 客户端库生成和消费消息，这些客户端库使用 kafka 作为消息代理提供不同的语言版本.

有什么建议吗?

解决方案

您可以使用 Kafka 源连接器，它将数据从 Mosquitto 等 MQTT 代理流式传输到 Kafka 集群.参见 https://github.com/evokly/kafka-connect-mqtt>

运行连接器的最简单方法是在独立模式下，单个实例将在单个节点上的 Kafka 集群上运行.您也可以在分布式模式下运行它(尽管配置更多)，这将在集群中分布连接器以获得更大的吞吐量.在分布式模式中，您可以设计允许水平扩展、并行吞吐量和高可用性的拓扑.实现额外的保证需要额外的负载平衡器、多个 MQTT 代理以及最后遗嘱和遗嘱场景来处理连接器崩溃，但这可能超出了这个问题的范围.

使用连接器方法具有 Kafka 集群的优势，可确保您的连接器处于活动状态并在必要时重新启动它.分布式模式提供更多优势.

How can we use mqtt protocol with kafka as a message broker?

The clients(android/ios/desktop java apps etc) will be producing and consuming messages using mqtt phao client side libraries which are available in different languages using kafka as a message broker.

Any advice?

解决方案

You can use a Kafka source connector which will stream data from an MQTT broker like Mosquitto into a Kafka cluster. See https://github.com/evokly/kafka-connect-mqtt

The simplest way to run the connector is in standalone mode, where a single instance will be running on the Kafka cluster on a single node. You can also run it in distributed mode (albeit with much more configuration) and this will distribute the connector across the cluster for greater throughput. In the distributed mode you can devise a topology that allows horizontal scaling, parallel throughput and high availability. Implementing additional guarantees requires additional load balancers, multiple MQTT brokers and last will and testament scenarios to deal with connectors crashing, but this is probably out of scope for this question.

Using the connector approach has the advantage of the Kafka cluster making sure your connector is alive and re-starting it if necessary. Distributed mode offers even more advantages.

这篇关于使用 mqtt 协议和 kafka 作为消息代理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！