卡夫卡有条件生产/消费 [英] Kafka conditional producing/consuming

问题描述

我有这种情况:

• 微服务MS1有一个数据库，假设有N条记录，并且一个外部源提供了新数据，如果有效，则应将其持久化

• microservice MS1 has a db, let's say with N records, and an external source provides new data that, if valids, should be persisted

验证过程由作为kafka主题T1的使用者的微服务MS2执行，MS1在其上发送新数据

validation process is performed by a microservice MS2 that is a consumer of a kafka topic T1, on which MS1 sends the new data

可能的N + 1条记录的验证涉及对所有先前的N条记录的查询.如果验证成功，则MS2会在主题T2上产生结果，MS1是该主题的使用者，因此它可以保留新的有效数据.

the validation of the potential N+1 record involves query on all the previous N records. If the validation is successful MS2 produces the result on topic T2, on which MS1 is consumer,so it can persist the new,valid,data.

问题出在下面.

想象一下，N + 1个有效的新数据太大，需要很多时间才能在db上写入:潜在 N + 2记录的验证失败可能是因为查询db只有N条可用记录，而不是N + 1条.

Imagine that the N+1 valid new data is too big and needs a lot of time to be writed on db: it can happen that validation of potential N+2 record fails because querying the db there are only N available record, instead of N+1.

是否有可能以某种方式使用kafka功能来暂停验证过程，直到MS1在db上提交先前的有效数据为止?

Is it possible, using kafka capabilities in some way, to pause the validation process until MS1 has committed the previous valid data on the db?

不使用kafka，我发现确保对最新db进行验证查询的唯一方法是在两个微服务之间进行rest调用，以便等待彼此的响应.

Not using kafka, the only way that i have found to ensure validation queries on a up-to-date db is to make rest call between the two microservices, so to wait the each other responses.

任何帮助或新的解决方案将不胜感激.

Any help or new solution would be really appreciated.

谢谢！

推荐答案

根据您在此处提供的信息，我认为您有3种选择:

Based on your information that you have provided here I think you have 3 options what you could do here:

1. 将MS2验证逻辑复制到MS1

1. Duplicate the MS2 validation logic to MS1

将MS2验证逻辑提取到共享库

Extract the MS2 validation logic to shared library

直接通过Rest呼叫从MS1呼叫MS2微服务

Call MS2 micro-service from MS1 directly with a Rest call

---

选项说明:

1. 验证逻辑是业务逻辑的重要组成部分之一，应始终是验证实体/集合的同一微服务的一部分.起初，复制听起来很奇怪，甚至可能是错误的，但在这种情况下，与在2个微服务中进行复制相比，它可以节省大量的开销.这样做，您只需重新部署MS1(即可从上方回答您的评论).请记住，否则这两个微服务之间的网络通信将给您带来许多问题，例如:性能，延迟，网络故障，分布式事务和其他.

1. Validation logic is one of the essential parts of the business logic and should always be part of the same micro-service which Entities/Aggregates are validated. At first duplication sounds strange and maybe even wrong but in this case it saves you a lot of overhead compared to having it in 2 micro-services. Doing this you would only need to redeploy your MS1(to answer your comment from above). Keep in mind that otherwise network communication between the 2 micro-services will bring you a lot of problems like: performance, delays, network failures, distributed transactions and others.

将常见的共享验证逻辑提取到某种库中(例如，在.NET nuget中，在Node.js npm中...)，并将其作为包包含在您的微服务MS1中也是一种选择.这样，您将不会在所有微服务中复制代码，但是如果您对某些验证的业务需求发生了变化，则将需要对所有微服务进行更改.如果发生这种情况，您将需要重新整理库和/或将一些代码提取回原始微服务.

Extracting the common shared validation logic to some sort of library(for example in .NET nuget, in Node.js npm...) and including it as package in your micro-service MS1 is also an option. This way you will not duplicate the code in all micro-services but if your business requirement for some validation changes you will need to have the change for all micro-services. If that happens you will need to rework your library and/or extract some of the code back to the originator micro-service.

可以直接使用REST API从MS1调用MS2 命令或查询方法(请阅读此答案，以了解微服务之间的通信方式).通过REST从另一个服务调用一个微服务没有错，在某些情况下，您别无选择.仍然要记住，调用另一个API进行验证时会出现一些延迟.

Calling MS2 from MS1 directly using REST API can be done over a Command or Query approach(please read this answer for the explanation on ways of communication between micro-services). There is nothing wrong in calling one micro-service from other over REST and in some cases you have no other choice. Still keep in mind that you will have some delays as you call another api to do your validation.

结论

我强烈建议您使用1.选项，并将验证逻辑从MS2复制到MS1.验证是域逻辑的一部分，并且应该在您拥有该逻辑的微服务中.这种或类似的验证逻辑可能适用于其他微服务，但是每个微服务都应该有自己的实现.选项2.主要用于某些业务不可知的事物，例如通用的测试基础结构，通用的数据库访问(存储库类)，通用的通信机制等.如果您想调用其他微服务以获取一些信息或发送某些命令(例如，为我保留此产品或类似产品)，则选项3.更好，但是对于验证，我会避免这样做.绝对有必要进行额外的网络通话以进行验证.

I would strongly recommend you to use the 1. option and duplicate your validation logic from MS2 to MS1. Validation is part of the Domain logic and should be in the same micro-service for which you have the logic. It can happen that this or similar validation logic is applicable for other micro-services but each micro-service should have its own implementation. The option 2. is mostly used for some business agnostic things like common test infrastructure, common database access(repository classes), common communication mechanism and so on. The option 3. is better if you want to call other micro-service to get some information or to send some command like(Reserve this Product for me or similar), but for doing validation I would avoid that. There is absolutely need for an extra network call just to do validation.

这篇关于卡夫卡有条件生产/消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！