Kafka Streams:RocksDB TTL [英] Kafka Streams: RocksDB TTL

问题描述

我了解默认TTL设置为无穷大(非正数).但是，如果我们需要将数据保留在存储中最多2天，是否可以使用RocksDBConfigSetter接口实现(即options.setWalTtlSeconds(172800))进行覆盖?还是会与Kafka Streams内部构件冲突?

I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals?

参考: https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config

推荐答案

目前无法实现.由于各种技术原因，Kafka Streams以硬编码方式禁用了RocksDB的TTL功能.还有一张票证: https://issues.apache.org/jira/浏览/KAFKA-4212

This is currently not possible. Kafka Streams disables RocksDB's TTL feature in a hard-coded way for various technical reasons. There is also a ticket for this: https://issues.apache.org/jira/browse/KAFKA-4212

要知道，您可以在2天后使用窗口存储来使旧记录过期.即，您要执行 stream.groupByKey().windowedBy(...).reduce(...)，其中 TimeWindow 为1ms，并使用虚拟"来减少只需返回密钥的最新值即可.

For know, you could use a windowed store to expire old record after 2 days. Ie, you do a stream.groupByKey().windowedBy(...).reduce(...) with a TimeWindow of 1ms and a "dummy" reduce that just return the latest value for a key.

这篇关于Kafka Streams:RocksDB TTL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！