Kafka:从 websockets 消费数据并推送到主题的连接器 [英] Kafka: Connector to consume data from websockets and push to topic

问题描述

我们有一个数据流管道，其中日志从 websocket 端点发送，需要在执行简单的数据增强(密码屏蔽等)后推送到 Splunk.

We have a data flow pipeline where logs are sent from a websocket endpoint, which need to be pushed to Splunk after doing simple data enhancing (password masking etc).

我正在检查 Kafka 是否可以用于此目的，因为容量非常高.所以，可能的流程是:

I was checking if Kafka can be used for this because the volumes are really high. So, the possible flow is:

Websocket Endpoint ---------some-wss-connector--------> Kafka Topic --------splunk-connector----------> Splunk

Websocket Endpoint ---------some-wss-connector--------> Kafka Topic --------splunk-connector----------> Splunk

我在以下位置找到了用于推送到 Splunk 的连接器:https://github.com/splunk/kafka-connect-splunk 并且效果很好.

I found the connector for pushing to Splunk at: https://github.com/splunk/kafka-connect-splunk and it works good.

需要其他连接器(上面粗体)的帮助才能从 Kafka 中的 websocket 端点读取数据并推送到主题.请问有人做过类似的吗?请指教.

Need help in the other connector (bolded above) to read from websocket endpoints in Kafka and push to a topic. Has anyone worked on something similar please? Please advice.

注意:我查看了 WebSocket 到 Kafka Topic Sc​​ala API但那是使用 Akka(与 Scala)，但我担心 Akka/Scala 目前不是我们企业的一部分.

NOTE: I've looked at WebSocket to Kafka Topic Scala API but that's using Akka (with Scala) but I am afraid Akka/Scala is not part of our enterprise at this moment.

提前致谢.维奈

推荐答案

好吧，我猜您需要某种适配器应用程序(无论是否采用 kafka connect 的形式).我的想法是编写一个实现 websocket 客户端的小型 Spring Boot 应用程序，请参阅此链接以了解如何(向下滚动一点):https://www.sitepoint.com/implementing-spring-websocket-服务器和客户端/

Well, I guess you´ll need some sort of adapter application (be it in the form of kafka connect or not). My idea is to write a small spring boot application which implements a websocket client, see this link to get an idea of how (scroll a bit down): https://www.sitepoint.com/implementing-spring-websocket-server-and-client/

然后，您可以使用 spring kafka 将通过 websocket 收到的消息推送到 apache kafka，这在我的经验中工作正常:https://spring.io/projects/spring-kafka

Then after that, you can push the messages received via websocket to apache kafka using spring kafka, which works fine in my experience: https://spring.io/projects/spring-kafka

对于从 SpringBoot 应用程序到 Kafka 的序列化，您可以使用 JSON，但是我建议使用 AVRO 和架构注册表的完整功能:https://docs.confluent.io/current/schema-registry/index.html

For Serialization from SpringBoot App to Kafka you can user JSON, however I recommmend the full bells and whistles with AVRO and a schema registry: https://docs.confluent.io/current/schema-registry/index.html

这个想法的主要缺点是您需要实施、维护、部署和扩展另一个应用程序.

The main drawback of this idea is the fact that you get another application you need to implement, maintain, deploy and scale.

希望有帮助！

这篇关于Kafka:从 websockets 消费数据并推送到主题的连接器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！