Kafka:连接器使用websockets中的数据并推送到主题 [英] Kafka: Connector to consume data from websockets and push to topic
问题描述
我们有一个数据流管道,其中日志是从websocket端点发送的,需要在进行简单的数据增强(密码屏蔽等)之后将其推送到Splunk.
We have a data flow pipeline where logs are sent from a websocket endpoint, which need to be pushed to Splunk after doing simple data enhancing (password masking etc).
我当时正在检查是否可以将Kafka用于此,因为数量确实很高.因此,可能的流程是:
I was checking if Kafka can be used for this because the volumes are really high. So, the possible flow is:
Websocket端点--------- some-wss-connector --------> Kafka主题------- splunk连接器 ----------> Splunk
Websocket Endpoint ---------some-wss-connector--------> Kafka Topic --------splunk-connector----------> Splunk
我在以下位置找到了用于推送到Splunk的连接器: https://github.com/splunk/kafka-connect-splunk ,效果很好.
I found the connector for pushing to Splunk at: https://github.com/splunk/kafka-connect-splunk and it works good.
需要其他连接器的帮助(如上所示),以从Kafka中的websocket端点中读取并推送至主题.请问有人做过类似的事情吗?请指教.
Need help in the other connector (bolded above) to read from websocket endpoints in Kafka and push to a topic. Has anyone worked on something similar please? Please advice.
注意:我看过 WebSocket到Kafka主题Scala API 但这使用的是Akka(与Scala一起使用),但恐怕Akka/Scala目前不在我们的企业范围内.
NOTE: I've looked at WebSocket to Kafka Topic Scala API but that's using Akka (with Scala) but I am afraid Akka/Scala is not part of our enterprise at this moment.
先谢谢了.威纳(Vinay)
Thanks in advance. Vinay
推荐答案
好吧,我想您需要某种适配器应用程序(无论它是否为kafka connect形式).我的想法是编写一个实现websocket客户端的小型spring boot应用程序,请参阅此链接以了解操作方式(向下滚动): https://www.sitepoint.com/implementing-spring-websocket-服务器和客户端/
Well, I guess you´ll need some sort of adapter application (be it in the form of kafka connect or not). My idea is to write a small spring boot application which implements a websocket client, see this link to get an idea of how (scroll a bit down): https://www.sitepoint.com/implementing-spring-websocket-server-and-client/
然后,您可以使用spring kafka将通过websocket接收到的消息推送到apache kafka,根据我的经验,它工作得很好: https://spring.io/projects/spring-kafka
Then after that, you can push the messages received via websocket to apache kafka using spring kafka, which works fine in my experience: https://spring.io/projects/spring-kafka
对于从SpringBoot App到Kafka的序列化,您可以 使用JSON,但是我建议使用AVRO和模式注册表进行全面介绍:
For Serialization from SpringBoot App to Kafka you can user JSON, however I recommmend the full bells and whistles with AVRO and a schema registry: https://docs.confluent.io/current/schema-registry/index.html
该想法的主要缺点是,您需要获得另一个应用程序来实现,维护,部署和扩展.
The main drawback of this idea is the fact that you get another application you need to implement, maintain, deploy and scale.
希望有帮助!
这篇关于Kafka:连接器使用websockets中的数据并推送到主题的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!