Kafka:连接器使用websockets中的数据并推送到主题 [英] Kafka: Connector to consume data from websockets and push to topic

问题描述

我们有一个数据流管道，其中日志是从websocket端点发送的，需要在进行简单的数据增强(密码屏蔽等)之后将其推送到Splunk.

We have a data flow pipeline where logs are sent from a websocket endpoint, which need to be pushed to Splunk after doing simple data enhancing (password masking etc).

我当时正在检查是否可以将Kafka用于此，因为数量确实很高.因此，可能的流程是:

I was checking if Kafka can be used for this because the volumes are really high. So, the possible flow is:

Websocket端点--------- some-wss-connector --------> Kafka主题------- splunk连接器 ----------> Splunk

Websocket Endpoint ---------some-wss-connector--------> Kafka Topic --------splunk-connector----------> Splunk

我在以下位置找到了用于推送到Splunk的连接器: https://github.com/splunk/kafka-connect-splunk ，效果很好.

I found the connector for pushing to Splunk at: https://github.com/splunk/kafka-connect-splunk and it works good.

需要其他连接器的帮助(如上所示)，以从Kafka中的websocket端点中读取并推送至主题.请问有人做过类似的事情吗?请指教.

Need help in the other connector (bolded above) to read from websocket endpoints in Kafka and push to a topic. Has anyone worked on something similar please? Please advice.

注意:我看过 WebSocket到Kafka主题Scala API 但这使用的是Akka(与Scala一起使用)，但恐怕Akka/Scala目前不在我们的企业范围内.

NOTE: I've looked at WebSocket to Kafka Topic Scala API but that's using Akka (with Scala) but I am afraid Akka/Scala is not part of our enterprise at this moment.

先谢谢了.威纳(Vinay)

Thanks in advance. Vinay

推荐答案

好吧，我想您需要某种适配器应用程序(无论它是否为kafka connect形式).我的想法是编写一个实现websocket客户端的小型spring boot应用程序，请参阅此链接以了解操作方式(向下滚动): https://www.sitepoint.com/implementing-spring-websocket-服务器和客户端/

Well, I guess you´ll need some sort of adapter application (be it in the form of kafka connect or not). My idea is to write a small spring boot application which implements a websocket client, see this link to get an idea of how (scroll a bit down): https://www.sitepoint.com/implementing-spring-websocket-server-and-client/

然后，您可以使用spring kafka将通过websocket接收到的消息推送到apache kafka，根据我的经验，它工作得很好: https://spring.io/projects/spring-kafka

Then after that, you can push the messages received via websocket to apache kafka using spring kafka, which works fine in my experience: https://spring.io/projects/spring-kafka

对于从SpringBoot App到Kafka的序列化，您可以 使用JSON，但是我建议使用AVRO和模式注册表进行全面介绍:

For Serialization from SpringBoot App to Kafka you can user JSON, however I recommmend the full bells and whistles with AVRO and a schema registry: https://docs.confluent.io/current/schema-registry/index.html

该想法的主要缺点是，您需要获得另一个应用程序来实现，维护，部署和扩展.

The main drawback of this idea is the fact that you get another application you need to implement, maintain, deploy and scale.

希望有帮助！

这篇关于Kafka:连接器使用websockets中的数据并推送到主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！