Kafka不删除带有墓碑的钥匙 [英] Kafka not deleting key with tombstone

问题描述

我使用以下属性创建一个kafka主题

I create a kafka topic with below properties

min.cleanable.dirty.ratio = 0.01，delete.retention.ms = 100，segment.ms = 100，cleanup.policy = compact

假设我按顺序插入k-v对 1111:1，1111:2，1111:null，2222:1 现在发生的是除最后一条消息外，日志压缩将在其余消息上运行并清除前两个消息，但保留 1111:null

Let's say I insert k-v pairs in order 1111:1, 1111:2, 1111:null, 2222:1 What happens now is except last message, the log compaction runs on rest of the messages and clears first two but retains 1111:null

访问文档

Kafka log compaction also allows for deletes. A message with a key and a null payload acts like a tombstone, a delete marker for that key. Tombstones get cleared after a period.

因此，我希望在实现delete.retention.ms时，空标记应使用键 1111

So, I am hoping when delete.retention.ms is achieved, the null marker should delete the message with key 1111

我有两个问题-为何墓碑标记不起作用?为什么从压缩中忽略最后一条消息?

I have two questions - Why is the tombstone marker not working? Why is the last message ignored from compaction?

这是server.properties文件的内容-

This is what server.properties file has -

log.retention.ms=100
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=100
log.cleaner.delete.retention.ms=100
log.cleaner.enable=true
log.cleaner.min.cleanable.ratio=0.01

推荐答案

逻辑删除记录的设计保留时间更长.原因是，经纪人没有追踪消费者.假设消费者在读取第一条记录后离线了一段时间.当使用者关闭时，日志压缩开始了.如果日志压缩将删除逻辑删除记录，那么使用者将永远不会得知该记录已被删除的事实.如果使用者实现了高速缓存，则可能发生记录永远不会被删除的情况.因此，逻辑删除保留的时间更长，以允许脱机消费者接收所有逻辑删除以进行本地清理.

Tombstone records are preserved longer by design. The reason is, that brokers don't track consumers. Assume, that a consumer goes offline for some time after reading the first record. While the consumer is down, log compaction kicks. If log compaction would delete the tombstone record, the consumer would never learn about the fact, that the record was deleted. If the consumer implements a cache, it could happen that the record gets never deleted. Thus, tombstone are preserved longer to allow offline consumer to receive all tombstones for local cleanup.

只有在 delete.retention.ms 之后，墓碑才会被删除(默认值为1天).注意:这是主题级别的配置，并且没有针对它的代理级别的配置.因此，如果要更改配置，则需要按主题设置配置.

Tombstone will be deleted only after delete.retention.ms (default value is 1 day). Note: this is a topic level configuration and there is no broker level configuration for it. Thus, you need to set the config per topic if you want to change it.

这篇关于Kafka不删除带有墓碑的钥匙的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！