Kafka Streams stateStores 容错恰好一次? [英] Kafka Streams stateStores fault tolerance exactly once?

问题描述

我们正在尝试使用 Kafka Streams 实现重复数据删除服务.总体而言，它将使用它的 RocksDB 状态存储，以便在处理过程中检查现有密钥.

We're trying to achieve a deduplication service using Kafka Streams. The big picture is that it will use its rocksDB state store in order to check existing keys during process.

如果我错了，请纠正我，但为了使那些 stateStores 也具有容错性，Kafka 流 API 将透明地复制 Kafka 主题(称为更改日志)中 stateStore 中的值.这样，如果我们的服务崩溃，另一个服务将能够根据在 Kafka 中找到的 changeLog 重建它的 stateStore.

Please correct me if I'm wrong, but to make those stateStores fault tolerant too, Kafka streams API will transparently copy the values in the stateStore inside a Kafka topic ( called the change Log). That way, if our service falls, another service will be able to rebuild its stateStore according to the changeLog found in Kafka.

但它向我提出了一个问题，这个StateStore --> 变更日志"本身是否恰好是一次?我的意思是，当服务更新它的 stateStore 时，它​​也会以一次的方式更新变更日志......?如果服务崩溃，另一个将承担负载，但我们能确定它不会错过崩溃服务的 stateStore 更新吗?

But it raises a question to my mind, do this " StateStore --> changelog" itself is exactly once ? I mean, When the service will update its stateStore, it will update the changelog in an exactly once fashion too.. ? If the service crash, another one will take the load, but can we sure it won't miss a stateStore update from the crashing service ?

问候，

扬尼克

推荐答案

简短的回答是肯定的.

使用事务 - 原子多分区写入 - Kafka Streams 确保在执行偏移提交时，状态存储也被刷新到代理上的变更日志主题.以上操作都是原子操作，因此如果其中一个操作失败，应用程序将从前一个偏移位置重新处理消息.

Using transaction - Atomic multi-partition write - Kafka Streams insure, that when offset commit was performed, state store was also flashed to changelog topic on the brokers. Above operations are Atomic, so if one of them will failed, application will reprocess messages from previous offset position.

您可以在以下博客中阅读更多关于恰好一次语义https://www.confluent.io/blog/enabling-exactly-kafka-streams/.有部分:Kafka Streams 如何保证 Exactly-Once 处理.

You can read in following blog more about exactly once semantic https://www.confluent.io/blog/enabling-exactly-kafka-streams/. There is section: How Kafka Streams Guarantees Exactly-Once Processing.

这篇关于Kafka Streams stateStores 容错恰好一次?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！