使用 Kafka 作为(CQRS)事件存储.好主意? [英] Using Kafka as a (CQRS) Eventstore. Good idea?

问题描述

虽然我之前遇到过 Kafka，但我最近才意识到 Kafka 可能被用作(基于)CQRS、eventstore.

Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS, eventstore.

Kafka 支持的要点之一:

One of the main points that Kafka supports:

• 事件捕获/存储，当然是所有 HA.
• 发布/订阅架构
• 能够重播事件日志，让新订阅者能够在事后注册系统.

诚然，我不是 100% 精通 CQRS/事件源，但这似乎非常接近事件存储应该是什么.有趣的是:我真的找不到太多关于 Kafka 被用作事件存储的信息，所以也许我错过了一些东西.

Admittedly I'm not 100% versed into CQRS / Event sourcing but this seems pretty close to what an eventstore should be. Funny thing is: I really can't find that much about Kafka being used as an eventstore, so perhaps I am missing something.

那么，Kafka 缺少什么才能成为一个好的事件存储库?它会起作用吗?用它生产?对洞察力、链接等感兴趣

So, anything missing from Kafka for it to be a good eventstore? Would it work? Using it production? Interested in insight, links, etc.

基本上系统的状态是根据系统收到的交易/事件保存的，而不是通常保存系统的当前状态/快照.(将其视为会计中的总帐:所有交易最终加起来为最终状态)这允许进行各种很酷的事情，但只需阅读提供的链接即可.

Basically the state of the system is saved based on the transactions/events the system has ever received, instead of just saving the current state / snapshot of the system which is what is usually done. (Think of it as a General Ledger in Accounting: all transactions ultimately add up to the final state) This allows all kinds of cool things, but just read up on the links provided.

推荐答案

Kafka 旨在成为一个消息传递系统，它与事件存储有许多相似之处，但引用他们的介绍:

Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro:

Kafka 集群保留所有已发布的消息——无论它们是否已被消耗——一段可配置的时间.例如如果保留时间设置为两天，然后保留两天后消息发布后可供消费，之后它将被丢弃以释放空间.Kafka的表现是有效的相对于数据大小恒定，因此保留大量数据不是问题.

The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.

因此，虽然消息可能会无限期保留，但预计它们将被删除.这并不意味着您不能将其用作事件存储，但使用其他东西可能会更好.查看 EventStore 寻找替代方案.

So while messages can potentially be retained indefinitely, the expectation is that they will be deleted. This doesn't mean you can't use this as an event store, but it may be better to use something else. Take a look at EventStore for an alternative.

Kafka 文档:

事件溯源是一种应用程序设计风格，其中将状态更改记录为按时间顺序排列的记录序列.Kafka 对非常大的存储日志数据的支持使其成为以这种风格构建的应用程序的出色后端.

Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.

更新 2

使用 Kafka 进行事件溯源的一个问题是所需主题的数量.通常在事件溯源中，每个实体(例如用户、产品等)都有一个事件流(主题).这样，可以通过重新应用流中的所有事件来重构实体的当前状态.每个 Kafka 主题由一个或多个分区组成，每个分区在文件系统上存储为一个目录.随着znode数量的增加，ZooKeeper也会有压力.

UPDATE 2

One concern with using Kafka for event sourcing is the number of required topics. Typically in event sourcing, there is a stream (topic) of events per entity (such as user, product, etc). This way, the current state of an entity can be reconstituted by re-applying all events in the stream. Each Kafka topic consists of one or more partitions and each partition is stored as a directory on the file system. There will also be pressure from ZooKeeper as the number of znodes increases.

这篇关于使用 Kafka 作为(CQRS)事件存储.好主意?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！