在网页上发送 Apache Kafka 数据 [英] Sending Apache Kafka data on web page

查看:17
本文介绍了在网页上发送 Apache Kafka 数据的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在构建一个实时能源监控系统,其中数据来自传感器.每一秒都会有新的数据.使用的数据将被聚合以呈现为图表.我研究了大量数据的实时流处理,它使我想到了 Apache Kafka.

I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka.

现在我的网络应用程序正在使用 Express js.我正在使用 kafka-node 库.现在,我以生产者的身份通过命令行手动插入新数据.在我的服务器代码中,我设置了一个 消费者 来监听 Topic1.

Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer. In my server code, I have set-up a consumer that listens to Topic1.

服务器代码:

var express = require('express');
var app = express();
var http = require('http').Server(app);
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: false });

var server = app.listen(3001, ()=>{
  console.log("app started on port 3001");
});

var io = require('socket.io').listen(server);


var kafka = require('kafka-node');

let Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(client,
      [
        {topic: 'Topic1', partition: 0}
      ],
      {
        autoCommit: false
      }
  );

app.use(express.static('public'));

consumer.on('message', (message) => {
  console.log(message.value);
  callSockets(io, message.value);
});

function callSockets(io, message){
  io.sockets.emit('update', message);
}

客户端代码:

<script type="text/javascript">
  var socket = io('http://localhost:3001');

  socket.on('connect', ()=>{
    console.log("connected");
  })

  socket.on('update', (data) =>{
    console.log(data);
  })
</script>

我正在使用 socket.io 来发出在 Kafka 中使用的消息.还有其他方法可以将Kafka数据发送到客户端吗?在我看来,使用 socket.io 在这里不太优雅.我是否以正确的方式接近它?欢迎任何建议!

I am using socket.io to emit the message consumed in Kafka. Is there any other way to send Kafka data to client side? It just seems to me using socket.io is not too elegant here. Have I approached it the right way? Any recommendations welcome!

谢谢.

推荐答案

在没有专门讨论您的解决方案的情况下,我认为您正在做正确的事情,即为客户端创建专用 API 以从中读取数据.客户端需要能够从某处接收更新.唯一更快"的方法是允许客户端直接从 Kafka 中提取数据,这会大大增加风险,因为 Kafka 并非设计为可公开访问的.

Without speaking to your solution specifically, I think you're doing the right thing creating a dedicated API for the client to read data from. The client needs to be able to receive updates from somewhere. The only "faster" way would be to allow the client to directly pull from Kafka which would increase risk substantially as Kafka is not designed to be publicly accessible.

因此,与需要更多代码的 C# 或 Java 中做同样的事情相比,您使用 node.js 的解决方案实际上相当优雅.

So your solution with node.js is actually rather elegant compared to doing the same thing in C# or Java which would require a lot more code.

这篇关于在网页上发送 Apache Kafka 数据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆