微服务客户端确认和事件溯源 [英] Microservices client acknowledgement and Event Sourcing

问题描述

场景

我正在使用微服务构建快递服务系统.我不确定一些事情，这是我的场景

I am building courier service system using Microservices. I am not sure of few things and here is my Scenario

1. Booking API - 这是客户下订单的地方
2. Payment API - 这是我们根据预订处理付款的地方
3. Notification API - 服务负责在一切完成后发送通知.

系统使用事件驱动架构.当客户下预订订单时，我在预订 API 中提交本地事务并发布事件.支付 API 和通知 API 订阅了它们各自的事件.一旦完成付款和通知 API 需要确认回预订 API.

The system is using event-driven Architecture. When customer places booking order , i commit local transaction in booking API and publish event. Payment API and notification API are subscribed to their respective event . Once Done Payment and notification API need to acknowledge back to Booking API.

我的问题是

发布事件后，我的预订服务无法阻止呼叫并返回到客户端(前端).我的客户端应用程序将如何检查交易状态或知道交易已完成?它每隔几秒轮询一次吗?由于这是分布式事务，任何服务都可能出现故障并且无法回复.在这种情况下，我的客户(前端)如何知道，因为它将继续等待.我正在考虑分布式事务的传奇.

After publishing the event my booking service can't block the call and goes back to the client (front end). How does my client app will have to check the status of transaction or it would know that transaction is completed? Does it poll every couple of seconds ? Since this is distributed transaction and any service can go down and won't be able to acknowledge back . In that case how do my client (front end) would know since it will keep on waiting. I am considering saga for distributed transactions.

实现这一切的最佳方式是什么?

What's the best way to achieve all of this ?

事件溯源

我想实施事件溯源来跟踪预订订单的完整跟踪.我是否必须在我的预订 API 中使用事件存储来实现这一点?或者事件存储在服务之间共享，因为我应该捕获来自不同服务的所有事件.实现这一点的最佳方法是什么?

I want to implement Event sourcing to track the complete track of the booking order. Does i have to implement this in my booking API with event store ? Or event store are shared between services since i am supposed to catch all the events from different services . What's the best way to implement this ?

非常感谢，

推荐答案

我想象的方式如下(受 Martin Kleppmann 的演讲影响 这里 和 这里).

The way I visualize this is as follows (influenced by Martin Kleppmann's talk here and here).

1. 最终用户下订单.订单写入 Kafka 主题.由于Kafka具有日志结构化存储，订单明细会在最短的时间内保存.这是一个原子操作(ACID"中的A")——要么全有，要么全无
2. 现在，只要用户下订单，用户就会想回读它(read-your-write).为了实现这一点，我们也可以将订单数据写入分布式缓存中.虽然双重写入通常不是一个好主意，因为它可能会导致部分失败(例如，写入 Kafka 成功，但写入缓存失败)，我们可以通过确保 Kafka 消费者之一将数据写入数据库来降低这种风险.因此，即使在罕见的缓存失败情况下，用户最终也可以从 DB 中读回数据.
3. 创建订单时缓存中订单的状态为进行中"
4. 然后使用一个或多个 kafka 消费者组来处理如下事件:支付和通知处理妥当，最终状态将写回缓存和数据库

5. 一个单独的 Kafka 消费者将收到来自支付和通知 API 的响应，并将更新写入缓存、数据库和网络套接字

1. The end user places an order. The order is written to a Kafka topic. Since Kafka has a log structured storage, the order details will be saved in the least possible time. It's an atomic operation ('A' in 'ACID') - all or nothing
2. Now as soon as the user places the order, the user would like to read it back (read-your-write). To acheive this we can write the order data in a distributed cache as well. Although dual write is not usually a good idea as it may cause partial failure (e.g. writing to Kafka is successful, but writing to cache fails), we can mitigate this risk by ensuring that one of the Kafka consumer writes the data in a database. So, even in a rare scenario of cache failure, the user can read the data back from DB eventually.
3. The status of the order in the cache as written at the time of order creation is "in progress"
4. One or more kafka consumer groups are then used to handle the events as follows: the payment and notification are handled properly and the final status will be written back to the cache and database

5. A separate Kafka consumer will then receive the response from the payment and notification apis and write the updates to cache, DB and a web socket

然后 websocket 将更新 UI 模型，然后更改将通过事件源反映在 UI 中.

The websocket will then update the UI model and the changes would be then reflected in the UI through event sourcing.

基于评论的进一步说明

1. 这里的基本思想是，我们使用流式传输为每个包含所需数据的服务构建缓存.例如账户服务需要来自支付和通知服务的反馈.因此，我们让这些服务将他们的响应写入一些 Kafka 主题，这些主题有一些消费者将响应写回订单服务的缓存

1. The basic idea here is that we build a cache using streaming for every service with data they need. For e.g. the account service needs feedback from the payment and notification services. Therefore, we have these services write their response to some Kafka topic which has some consumers that write the response back to order service's cache

基于 Kafka(或任何类似技术)的 ACID 属性，消息永远不会丢失.最终我们将得到全部或一无所有.这就是原子性.如果订单服务无法写入订单，则会以同步方式将错误响应发送回客户端，并且用户可能会在一段时间后重试.如果订单服务成功，则对其他服务的响应最终必须流回其缓存.如果其中一个服务宕机一段时间，响应会延迟，但最终会在服务恢复时发送

Based on the ACID properties of Kafka (or any similar technology), the message will never be lost. Eventually we will get all or nothing. That's atomicity. If the order service fails to write the order, an error response is sent back to the client in a synchronous way and the user probably retries after some time. If the order service is successful, the response to the other services must flow back to its cache eventually. If one of the services is down for some time, the response will be delayed, but it will be sent eventually when the service resumes

客户端不需要轮询.结果将通过使用 websocket 的流传播给它.UI页面会监听websocket，当消费者将反馈写入缓存中时，它也可以写入websocket.这将通知 UI.然后，如果您使用诸如 Angular 或 ReactJS 之类的东西，则可以使用在 websocket 接收到的值刷新 UI 的相应部分.在发生这种情况之前，用户一直看到状态进行中"，就像在创建订单时写入缓存一样，即使用户刷新页面，也会从缓存中检索相同的状态.如果缓存值过期并遵循 LRU 机制，将从数据库中获取相同的值并将其写回缓存以服务未来的请求.一旦其他服务的反馈可用，新结果将使用 websocket 流式传输.在页面刷新时，将从缓存或数据库中获得新状态

The clients need not poll. The result will be propagated to it through streaming using websocket. The UI page will listen to the websocket As the consumer writes the feedback in the cache, it can also write to the websocket. This will notify the UI. Then if you use something like Angular or ReactJS, the appropriate section of the UI can be refreshed with the value received at the websocket. Until that happens user keeps seeing the status "in progress" as was written to the cache at the time of order creation Even if the user refreshes the page, the same status is retrieved from the cache. If the cache value expires and follows a LRU mechanism, the same value will be fetched from the DB and wriitten back to the cache to serve future requests. Once the feedback from the other services are available, the new result will be streamed using websocket. On page refresh, new status would be available from the cache or DB

这篇关于微服务客户端确认和事件溯源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！