Kafka Streams:RocksDB TTL [英] Kafka Streams: RocksDB TTL
问题描述
我了解默认TTL设置为无穷大(非正数).但是,如果我们需要将数据保留在存储中最多2天,是否可以使用RocksDBConfigSetter接口实现(即options.setWalTtlSeconds(172800))进行覆盖?还是会与Kafka Streams内部构件冲突?
I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals?
推荐答案
目前无法实现.由于各种技术原因,Kafka Streams以硬编码方式禁用了RocksDB的TTL功能.还有一张票证: https://issues.apache.org/jira/浏览/KAFKA-4212
This is currently not possible. Kafka Streams disables RocksDB's TTL feature in a hard-coded way for various technical reasons. There is also a ticket for this: https://issues.apache.org/jira/browse/KAFKA-4212
要知道,您可以在2天后使用窗口存储来使旧记录过期.即,您要执行 stream.groupByKey().windowedBy(...).reduce(...)
,其中 TimeWindow
为1ms,并使用虚拟"来减少只需返回密钥的最新值即可.
For know, you could use a windowed store to expire old record after 2 days. Ie, you do a stream.groupByKey().windowedBy(...).reduce(...)
with a TimeWindow
of 1ms and a "dummy" reduce that just return the latest value for a key.
这篇关于Kafka Streams:RocksDB TTL的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!