消耗Kafka主题后发送HTTP响应 [英] Sending HTTP response after consuming a Kafka topic

问题描述

我目前正在编写一个包含大量微服务的Web应用程序.我目前正在探索如何在所有这些服务之间进行正确的通信，并且我决定坚持使用消息总线，或更具体地说是Apache Kafka.

但是，我有几个问题不确定如何从概念上解决. 我正在使用API​​网关服务作为应用程序的主要条目.它充当将操作转发到适用的微服务的主要代理. 请考虑以下情形:

1. 用户将一些信息发送到POST请求到API网关.
2. 网关会生成一条新消息，并将其发布到Kafka主题.
3. 订阅的微服务获取主题中的消息并处理数据.

那么，我现在应该如何从网关响应客户端?如果我需要该微服务的一些数据怎么办?这样的HTTP请求可能会超时.我应该在客户端和API网关之间坚持使用websockets吗?

而且，如果客户端发送GET请求以获取一些数据，我应该如何使用Kafka来解决该问题?

谢谢.

解决方案

假设您要创建一个订单.这应该是这样的:

1. 传统上，我们过去在RDBMS表中具有一个自动递增字段或一个序列来创建订单ID.但是，这意味着在我们将订单保存到数据库中之前不会生成订单ID.现在，在Kafka中写入数据时，我们不会立即写入数据库，并且Kafka无法生成订单ID.因此，您需要使用一些可扩展的id生成实用程序，例如Twitter Snowflake或具有类似体系结构的东西，以便甚至在用Kafka编写订单之前也可以生成订单id.

2. 一旦有了订单ID，就以原子方式(全有或全无)在Kafka主题上写一条事件消息.成功完成此操作后，您可以将成功响应发送回客户端.在此阶段不要写多个主题，因为写多个主题会失去原子性.您总是可以有多个将事件写入其他多个主题的使用者组.一个消费者组应该将数据写入某个持久性数据库中以进行查询

3. 您现在需要解决读自己写的问题，即在收到成功响应后，用户希望立即查看订单.但是您的数据库可能尚未使用订单数据进行更新.为此，请将订单数据写入Kafka之后并返回成功响应之前，立即将订单数据写入Redis或Memcached之类的分布式缓存中.当用户读取订单时，将返回缓存的数据

4. 现在，您需要使用最新的订单状态更新缓存.您可以始终让Kafka消费者从Kafka主题中读取订单状态

5. 为确保不需要将所有订单保留在缓存中.您可以基于LRU逐出数据.如果在读取订单时数据不在缓存中，将从数据库中读取数据并将其写入缓存以供将来请求

6. 最后，如果要确保为订单保留订购的商品，以便其他人都不能拿走，例如预订飞机座位或一本书的最后一本，则需要一种共识算法.您可以为此使用Apache Zookeeper，并在该项目上创建分布式锁

I’m currently writing a web application that has a bunch of microservices. I’m currently exploring how to properly communicate between all these services and I’ve decided to stick with a message bus, or more specifically Apache Kafka.

However, I have a few questions that I’m not sure how to conceptually get around. I’m using an API Gateway-service as the main entry to the application. It acts as the main proxy to forward operations to the applicable microservices. Consider the following scenario:

1. User sends a POST-request to the API Gateway with some information.
2. The Gateway produces a new message and publishes it to a Kafka topic.
3. Subscribed microservices pick up the message in the topic and processes the data.

So, how am I now supposed to respond to the client from the Gateway? What if I need some data from that microservice? Feels like that HTTP request could timeout. Should I stick with websockets between the client and API Gateway instead?

And also, if the client sends a GET request to fetch some data, how am I supposed to approach that using Kafka?

Thanks.

解决方案

Let's say you're going to create an order. This is how it should work:

1. Traditionally we used to have an auto-increment field or a sequence in the RDBMS table to create an order id. However, this means order id is not generated until we save the order in DB. Now, when writing data in Kafka, we're not immediately writing to the DB and Kafka cannot generate order id. Hence you need to use some scalable id generation utility like Twitter Snowflake or something with the similar architecture so that you can generate an order id even before writing the order in Kafka

2. Once you have the order id, write a single event message on Kafka topic atomically (all-or-nothing). Once this is successfully done, you can send back a success response to the client. Do not write to multiple topics at this stage as you'll lose atomicity by writing to multiple topics. You can always have multiple consumer groups that write the event to multiple other topics. One consumer group should write the data in some persistent DB for querying

3. You now need to address the read-your-own-write i.e. immediately after receiving success response the user would want to see the order. But your DB is probably not yet updated with the order data. To acheive this, write the order data to a distributed cache like Redis or Memcached immediately after writing the order data to Kafka and before returning the success response. When the user reads the order, the cached data is returned

4. Now you need to keep the cache updated with the latest order status. That you can always do with a Kafka consumer reading the order status from a Kafka topic

5. To ensure that you don't need to keep all orders in cache memory. You can evict data based on LRU. If while reading an order, the data is not on cache, it will be read from the DB and written to the cache for future requests

6. Finally, if you want to ensure that the ordered item is reserved for the order so that no one else can take, like booking a flight seat, or the last copy of a book, you need a consensus algorithm. You can use Apache Zookeeper for that and create a distribured lock on the item

这篇关于消耗Kafka主题后发送HTTP响应的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！