Kafka Streams:RocksDB TTL [英] Kafka Streams: RocksDB TTL

问题描述

我知道默认 TTL 设置为无穷大(非正).但是，如果我们需要在存储中保留最多 2 天的数据，我们是否可以使用 RocksDBConfigSetter 接口实现进行覆盖，即 options.setWalTtlSeconds(172800)?或者它会与 Kafka 流内部发生冲突吗?

I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals?

参考:https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config

推荐答案

这目前是不可能的.由于各种技术原因，Kafka Streams 以硬编码方式禁用了 RocksDB 的 TTL 功能.还有一张票:https://issues.apache.org/jira/浏览/KAFKA-4212

This is currently not possible. Kafka Streams disables RocksDB's TTL feature in a hard-coded way for various technical reasons. There is also a ticket for this: https://issues.apache.org/jira/browse/KAFKA-4212

现在，您可以使用窗口存储在 2 天后使旧记录过期.即，您执行一个 stream.groupByKey().windowedBy(...).reduce(...)，其中 TimeWindow 为 1ms 和一个dummy"减少只返回键的最新值.

For now, you could use a windowed store to expire old record after 2 days. Ie, you do a stream.groupByKey().windowedBy(...).reduce(...) with a TimeWindow of 1ms and a "dummy" reduce that just return the latest value for a key.

这篇关于Kafka Streams:RocksDB TTL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！