Kafka 有条件的生产/消费 [英] Kafka conditional producing/consuming
问题描述
我有这种情况:
微服务 MS1 有一个数据库，假设有 N 条记录，外部源提供新数据，如果有效，则应保留这些数据
microservice MS1 has a db, let's say with N records, and an external source provides new data that, if valids, should be persisted
验证过程由作为 kafka 主题 T1 的消费者的微服务 MS2 执行，MS1 在该主题上发送新数据
validation process is performed by a microservice MS2 that is a consumer of a kafka topic T1, on which MS1 sends the new data
潜在 N+1 记录的验证涉及对所有前 N 条记录的查询.如果验证成功，MS2 会在主题 T2 上产生结果，MS1 是消费者，因此它可以持久保存新的、有效的数据.
the validation of the potential N+1 record involves query on all the previous N records. If the validation is successful MS2 produces the result on topic T2, on which MS1 is consumer,so it can persist the new,valid,data.
问题如下.
想象N+1个有效的新数据太大，需要很多时间写在db上:可能会发生潜在 N+2记录的验证失败，因为查询db只有N条可用记录，而不是N+1条.
Imagine that the N+1 valid new data is too big and needs a lot of time to be writed on db: it can happen that validation of potential N+2 record fails because querying the db there are only N available record, instead of N+1.
是否有可能以某种方式使用 kafka 功能暂停验证过程，直到 MS1 提交了数据库上的先前有效数据?
Is it possible, using kafka capabilities in some way, to pause the validation process until MS1 has committed the previous valid data on the db?
不使用 kafka，我发现确保对最新数据库进行验证查询的唯一方法是在两个微服务之间进行休息调用，以便等待彼此的响应.
Not using kafka, the only way that i have found to ensure validation queries on a up-to-date db is to make rest call between the two microservices, so to wait the each other responses.
非常感谢任何帮助或新的解决方案.
Any help or new solution would be really appreciated.
谢谢！
推荐答案
根据您在此处提供的信息，我认为您有 3 个选项可以在此处执行:
Based on your information that you have provided here I think you have 3 options what you could do here:
将 MS2 验证逻辑复制到 MS1
Duplicate the MS2 validation logic to MS1
将 MS2 验证逻辑提取到共享库
Extract the MS2 validation logic to shared library
通过 Rest 调用直接从 MS1 调用 MS2 微服务
Call MS2 micro-service from MS1 directly with a Rest call
<小时>
选项说明:
验证逻辑是业务逻辑的重要组成部分之一，应该始终是验证实体/聚合的同一个微服务的一部分.起初，重复听起来很奇怪，甚至可能是错误的，但在这种情况下，与在 2 个微服务中进行复制相比，它为您节省了大量开销.这样做你只需要重新部署你的 MS1(回答你上面的评论).请记住，否则两个微服务之间的网络通信会给您带来很多问题，例如:性能、延迟、网络故障、分布式事务等.
Validation logic is one of the essential parts of the business logic and should always be part of the same micro-service which Entities/Aggregates are validated. At first duplication sounds strange and maybe even wrong but in this case it saves you a lot of overhead compared to having it in 2 micro-services. Doing this you would only need to redeploy your MS1(to answer your comment from above). Keep in mind that otherwise network communication between the 2 micro-services will bring you a lot of problems like: performance, delays, network failures, distributed transactions and others.
将通用共享验证逻辑提取到某种库(例如在 .NET nuget 中，在 Node.js npm 中...)并将其作为包包含在您的微服务 MS1 中也是一种选择.这样您就不会在所有微服务中复制代码，但是如果您的业务需求对某些验证发生变化，您将需要对所有微服务进行更改.如果发生这种情况，您将需要返工您的库和/或将一些代码提取回原始微服务.
Extracting the common shared validation logic to some sort of library(for example in .NET nuget, in Node.js npm...) and including it as package in your micro-service MS1 is also an option. This way you will not duplicate the code in all micro-services but if your business requirement for some validation changes you will need to have the change for all micro-services. If that happens you will need to rework your library and/or extract some of the code back to the originator micro-service.
可以通过 REST API 直接从 MS1 调用 MS2命令或查询方法(请阅读此答案以了解微服务之间的通信方式).通过 REST 从其他微服务调用一个微服务并没有错，在某些情况下，您别无选择.仍然请记住，当您调用另一个 api 进行验证时，您会遇到一些延迟.
Calling MS2 from MS1 directly using REST API can be done over a Command or Query approach(please read this answer for the explanation on ways of communication between micro-services). There is nothing wrong in calling one micro-service from other over REST and in some cases you have no other choice. Still keep in mind that you will have some delays as you call another api to do your validation.
结论
我强烈建议您使用 1. 选项并将您的验证逻辑从 MS2 复制到 MS1.验证是域逻辑的一部分，应该在您拥有逻辑的同一个微服务中.这种或类似的验证逻辑可能适用于其他微服务，但每个微服务都应该有自己的实现.选项 2. 主要用于一些业务不可知的事情，如公共测试基础设施、公共数据库访问(存储库类)、公共通信机制等.如果您想调用其他微服务来获取一些信息或发送一些命令(为我或类似的人保留此产品)，则选项 3. 更好，但为了进行验证，我会避免这样做.绝对需要额外的网络调用来进行验证.
I would strongly recommend you to use the 1. option and duplicate your validation logic from MS2 to MS1. Validation is part of the Domain logic and should be in the same micro-service for which you have the logic. It can happen that this or similar validation logic is applicable for other micro-services but each micro-service should have its own implementation. The option 2. is mostly used for some business agnostic things like common test infrastructure, common database access(repository classes), common communication mechanism and so on. The option 3. is better if you want to call other micro-service to get some information or to send some command like(Reserve this Product for me or similar), but for doing validation I would avoid that. There is absolutely need for an extra network call just to do validation.
这篇关于Kafka 有条件的生产/消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
