Kafka Streams state只能一次存储容错能力吗? [英] Kafka Streams stateStores fault tolerance exactly once?

问题描述

我们正在尝试使用Kafka Streams实现重复数据删除服务. 总体情况是，它将使用其rocksDB状态存储来在处理过程中检查现有密钥.

We're trying to achieve a deduplication service using Kafka Streams. The big picture is that it will use its rocksDB state store in order to check existing keys during process.

如果我错了，请纠正我，但是为了使这些stateStores也容错，Kafka Streams API将透明地复制Kafka主题内的stateStore中的值(称为更改日志). 这样，如果我们的服务失败了，另一个服务将能够根据在Kafka中找到的changeLog重建其stateStore.

Please correct me if I'm wrong, but to make those stateStores fault tolerant too, Kafka streams API will transparently copy the values in the stateStore inside a Kafka topic ( called the change Log). That way, if our service falls, another service will be able to rebuild its stateStore according to the changeLog found in Kafka.

但是这引起了我的疑问，这个"StateStore-> changelog"本身是否只是一次? 我的意思是，当服务将更新其stateStore时，它也将以完全一次的方式更新changelog. 如果服务崩溃，那么另一项将承担负载，但是我们可以确定它不会丢失崩溃服务中的stateStore更新吗?

But it raises a question to my mind, do this " StateStore --> changelog" itself is exactly once ? I mean, When the service will update its stateStore, it will update the changelog in an exactly once fashion too.. ? If the service crash, another one will take the load, but can we sure it won't miss a stateStore update from the crashing service ?

此致

Yannick

推荐答案

简单答案是肯定的.

使用事务-原子多分区写入-Kafka Streams确保在执行偏移提交时，状态存储也被刷新到代理上的changelog主题.上述操作是原子操作，因此，如果其中之一失败，则应用程序将从前一个偏移位置重新处理消息.

Using transaction - Atomic multi-partition write - Kafka Streams insure, that when offset commit was performed, state store was also flashed to changelog topic on the brokers. Above operations are Atomic, so if one of them will failed, application will reprocess messages from previous offset position.

您可以在以下博客中详细了解一次语义 https://www.confluent.io/blog/enabling-exactly-kafka-streams/.有以下部分: How Kafka Streams Guarantees Exactly-Once Processing .

You can read in following blog more about exactly once semantic https://www.confluent.io/blog/enabling-exactly-kafka-streams/. There is section: How Kafka Streams Guarantees Exactly-Once Processing.

这篇关于Kafka Streams state只能一次存储容错能力吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！