在 kafka 事件驱动的主干中保持服务同步 [英] Keeping services in sync in a kafka event driven backbone

问题描述

假设我在系统设计中使用 Kafka 作为我所有微服务的事件驱动主干.许多微服务使用事件数据来填充其内部数据库.现在有一个需求，我需要创建一个新服务，它使用一些事件数据.该服务只能在它上线后才能使用事件，因此不会有很多丢失的数据.我想要一个策略，这样我就不必通过编写脚本来回填我的内部数据库.

Say I am using Kafka as the event-driven backbone for all my microservices in my system design. Many microservices use the events data to populate their internal databases. Now there is a requirement where I need to create a new service and it uses some events data. The service will only be able to consume events after the time it comes live and hence, won't have a lot of data that it missed. I want a strategy such that I don't have to backfill my internal databases by writing out scripts.

我可以采用哪些不会给 Kafka & 带来巨大负载的很酷的策略?没有在我创建的新服务中使用大量脚本来回填数据?

What are some cool strategies I can have which do not create a huge load on Kafka & does not account for a lot of scripting to backfill data in the new services that I ever create?

推荐答案

您可以在此处采用一些策略，具体取决于您将数据发布到 kafka 主题的方式.这里有一些想法:

There are a few strategies you can have here, depending on how you publish data to a kafka topic. Here are a few ideas:

• 首先，您可以将kafka主题的保留设置为永远，这意味着它将存储所有数据.这是可以的，因为 kafka 也是为此目的而构建的.请参阅这个.通过这样做，任何活跃的新服务都可以从一开始就开始消费数据.

• first, you can set the retention of a kafka topic to be forever, meaning that it will store all the data. This is OK as kafka is built for this purpose as well. See this. By doing this, any new service that come alive can start consuming data from the start.

如果你使用 kafka 来发布给定实体/聚合的最新状态，你也可以考虑 将主题配置为压缩.这将使您至少存储主题上实体/聚合的最新状态，并且开始侦听主题的新消费者将需要配置的数据更少.但是，您的消费者仍然需要知道如何为每个实体/聚合处理多条消息，因为您无法保证主题中只有一条消息.

if you are using kafka for latest state publishing for a given entity/aggregate, you can also consider configuring the topic to be a compacted. This will let you store at least the latest state of your entity/aggregate on the topic, and new consumers that starts listening on the topic will have less data to configure. However, your consumers still need to know how to process multiple messages per entity/aggregate as you cannot guarantee it will have exactly one message in the topic.

这篇关于在 kafka 事件驱动的主干中保持服务同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！