在kafka事件驱动的主干中保持服务同步 [英] Keeping services in sync in a kafka event driven backbone

问题描述

说我在系统设计中将Kafka用作所有微服务的事件驱动主干.许多微服务使用事件数据来填充其内部数据库. 现在有一个需求，我需要创建一个新服务，它使用一些事件数据.该服务仅在活动上线后才能使用事件，因此不会丢失很多数据.我想要一种策略，使我不必通过编写脚本来回填内部数据库.

Say I am using Kafka as the event-driven backbone for all my microservices in my system design. Many microservices use the events data to populate their internal databases. Now there is a requirement where I need to create a new service and it uses some events data. The service will only be able to consume events after the time it comes live and hence, won't have a lot of data that it missed. I want a strategy such that I don't have to backfill my internal databases by writing out scripts.

我可以采用哪些不错的策略，这些策略不会给Kafka&不会花很多脚本来回填我曾经创建的新服务中的数据?

What are some cool strategies I can have which do not create a huge load on Kafka & does not account for a lot of scripting to backfill data in the new services that I ever create?

推荐答案

根据您将数据发布到kafka主题的方式，这里可以采用几种策略.这里有一些想法:

There are a few strategies you can have here, depending on how you publish data to a kafka topic. Here are a few ideas:

• 首先，您可以将kafka主题的保留时间设置为永久，这意味着它将存储所有数据.这也可以，因为kafka也是为此目的而构建的.参见此.这样，任何启用的新服务都可以从一开始就使用数据.

• first, you can set the retention of a kafka topic to be forever, meaning that it will store all the data. This is OK as kafka is built for this purpose as well. See this. By doing this, any new service that come alive can start consuming data from the start.

如果您正在使用kafka进行给定实体/集合的最新状态发布，则还可以考虑

if you are using kafka for latest state publishing for a given entity/aggregate, you can also consider configuring the topic to be a compacted. This will let you store at least the latest state of your entity/aggregate on the topic, and new consumers that starts listening on the topic will have less data to configure. However, your consumers still need to know how to process multiple messages per entity/aggregate as you cannot guarantee it will have exactly one message in the topic.

这篇关于在kafka事件驱动的主干中保持服务同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！