在Lagom Microservice中提取流数据的最佳方法 [英] Best approach to ingest Streaming Data in Lagom Microservice
问题描述
我正在创建流分析应用程序，其中的每个分析/功能都将实现为微服务，以便该分析以后可以在其他项目中使用.
I am creating streaming analytics application in which each analytics/functionality will implement as a Microservice so that this analytics can able to use in the different project later.
我正在使用Lagom创建微服务.我是Lagom的新手，这就是为什么我遇到一些疑问的原因.
I am using Lagom for creating Microservice. I am new in lagom that's why i came across with some doubts.
我不知道将数据流(来自多个传感器)发送到微服务，然后该微服务将数据发布到kafka主题的最佳方法是什么.
I don't understand what will be the best approach to Post my stream of data (coming from multiple sensors) to microservice and then this microservice publish data to kafka topic.
-
服务描述中的流消息是否具有Lagom功能 ServiceCall [Source [String，NotUsed]，Source [String，NotUsed]] 从数百个wifi传感器发布数据流(大数据)的正确方法吗?它有租约来处理如此大量的接近实时(= 5秒)接收的数据流吗?
Does Lagom Feature of Stream Message in Service Description ServiceCall[ Source[String, NotUsed], Source[String, NotUsed]] is it the right approach to Post streams of data (Big Data) from hundreds of wifi sensors? Does it have a tenancy to deal with this huge amount of data streams receiving near real time (=5 Sec)?
第二，在将数据发布到kafka主题时，为什么我必须 实施持久性实体(由Lagom推荐)?因为卡夫卡 本身保证至少一次传递消息
secondaly, while publish data to kafka topics why I have to implement Persistent Entity (recommended by Lagom)? Because Kafka itself guarantees at-least-once delivery of message
我的应用程序不是CRUD应用程序，它仅支持处理流数据.
My application is not CRUD application, it only support to process streaming data.
推荐答案
-
Lagom的流式调用使用WebSockets.它基于Play的WebSocket支持，可以扩展到数百万个已连接的客户端.我不会将数百个wifi传感器称为海量数据，Lagom应该可以轻松处理它，并且Lagom也可以轻松地水平缩放，因此，如果您要处理的过程很繁重，则可以轻松地将处理过程分散到多个节点上.
Lagom's streaming calls use WebSockets. It's built on Play's WebSocket support, which can scale to millions of connected clients. I wouldn't call hundreds of wifi sensors a huge amount of data, Lagom should easily handle it, and Lagom can also easily be scaled horizontally, so if the processing you're doing is heavy, you can easily spread that processing across many nodes.
Lagom当前不支持将传入的WebSocket流发布到Kafka.尽管Kafka至少会至少保证一次将消息发布到Kafka ，但是在将消息首次发送给Kafka时并没有这样的保证.例如，如果您执行副作用，例如更新数据库，然后发布一条消息，则无法保证如果应用程序在数据库更新和将消息发布到Kafka之间崩溃时，该消息最终将发布给Kafka(实际上不会，该消息将会丢失).这就是为什么Lagom鼓励仅将数据库事件流发布到Kafka的原因，因为以这种方式发布事件日志确实可以确保随后需要发送到Kafka的任何数据库操作都至少发生一次.但是，如果您没有做副作用(听起来好像不是)，那么这可能与您无关.在这种情况下，我建议直接使用akka-streams-kafka(Lagom的Kafka支持所基于的东西).
Publishing an incoming WebSocket stream to Kafka is not currently supported in Lagom. While Kafka does guarantee at least once once a message is published to Kafka, there are no such guarantees when getting that message into Kafka in the first instance. For example, if you do a side effect, such as update a database, then publish a message, there's no guarantee that if the application crashes between when the database is updated, and when the message is published to Kafka, that that message will eventually be published to Kafka (in fact it won't be, that message will be lost). This is why Lagom encourages only database event streams to be published to Kafka, since publishing the event log in that way does guarantee that any database operation that then needs to be sent to Kafka does happen at least once. However, if you're not doing side effects, which it sounds like you're not, then this might not be relevant to you. What I would recommend in that case would be to use akka-streams-kafka (what Lagom's Kafka support is built on) directly.
在您的用例此处中，我提出了一个问题.
I've raised an issue referencing your use case here.
这篇关于在Lagom Microservice中提取流数据的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
