在网页上发送Apache Kafka数据 [英] Sending Apache Kafka data on web page

问题描述

我正在构建一个实时能源监控系统，其中的数据来自传感器.每秒将有新数据.所使用的数据将汇总为图表.我研究了使用大量数据进行实时流处理的方法，并将其带到了Apache Kafka.

I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka.

现在我的Web应用程序正在使用Express js.我正在使用 kafka-node 库.现在，我现在以生产者的形式通过命令行手动插入新数据.在服务器代码中，我设置了一个消费者来监听 Topic1 .

Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer. In my server code, I have set-up a consumer that listens to Topic1.

服务器代码:

var express = require('express');
var app = express();
var http = require('http').Server(app);
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: false });

var server = app.listen(3001, ()=>{
  console.log("app started on port 3001");
});

var io = require('socket.io').listen(server);


var kafka = require('kafka-node');

let Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(client,
      [
        {topic: 'Topic1', partition: 0}
      ],
      {
        autoCommit: false
      }
  );

app.use(express.static('public'));

consumer.on('message', (message) => {
  console.log(message.value);
  callSockets(io, message.value);
});

function callSockets(io, message){
  io.sockets.emit('update', message);
}

客户代码:

<script type="text/javascript">
  var socket = io('http://localhost:3001');

  socket.on('connect', ()=>{
    console.log("connected");
  })

  socket.on('update', (data) =>{
    console.log(data);
  })
</script>

我正在使用socket.io发出Kafka中消耗的消息.还有其他方法可以将Kafka数据发送到客户端吗?在我看来，在这里使用socket.io不太好用.我采取正确的方法了吗?欢迎任何建议！

I am using socket.io to emit the message consumed in Kafka. Is there any other way to send Kafka data to client side? It just seems to me using socket.io is not too elegant here. Have I approached it the right way? Any recommendations welcome!

谢谢.

推荐答案

在没有专门针对您的解决方案的情况下，我认为您做对了，为客户端读取数据创建了专用的API.客户端需要能够从某个地方接收更新.唯一更快"的方法是允许客户直接从Kafka撤出，这将大大增加风险，因为Kafka并非设计为公开可用.

Without speaking to your solution specifically, I think you're doing the right thing creating a dedicated API for the client to read data from. The client needs to be able to receive updates from somewhere. The only "faster" way would be to allow the client to directly pull from Kafka which would increase risk substantially as Kafka is not designed to be publicly accessible.

因此，与在C#或Java中执行相同操作(这需要更多代码)相比，使用node.js的解决方案实际上是相当优雅的.

So your solution with node.js is actually rather elegant compared to doing the same thing in C# or Java which would require a lot more code.

这篇关于在网页上发送Apache Kafka数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！