卡夫卡不删除带有墓碑的密钥 [英] Kafka not deleting key with tombstone

问题描述

我创建了一个具有以下属性的 kafka 主题

I create a kafka topic with below properties

min.cleanable.dirty.ratio=0.01,delete.retention.ms=100,segment.ms=100,cleanup.policy=compact

假设我按顺序插入 k-v 对 1111:1, 1111:2, 1111:null, 2222:1现在发生的情况是除了最后一条消息，日志压缩在其余消息上运行并清除前两条但保留 1111:null

Let's say I insert k-v pairs in order 1111:1, 1111:2, 1111:null, 2222:1 What happens now is except last message, the log compaction runs on rest of the messages and clears first two but retains 1111:null

根据文档，

Kafka log compaction also allows for deletes. A message with a key and a null payload acts like a tombstone, a delete marker for that key. Tombstones get cleared after a period.

所以，我希望当实现 delete.retention.ms 时，空标记应该删除带有 1111

So, I am hoping when delete.retention.ms is achieved, the null marker should delete the message with key 1111

我有两个问题 - 为什么墓碑标记不起作用?为什么最后一条消息会被压缩忽略?

I have two questions - Why is the tombstone marker not working? Why is the last message ignored from compaction?

这是 server.properties 文件的内容 -

This is what server.properties file has -

log.retention.ms=100
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=100
log.cleaner.delete.retention.ms=100
log.cleaner.enable=true
log.cleaner.min.cleanable.ratio=0.01

推荐答案

Tombstone 记录的设计保留时间更长.原因是，经纪人不跟踪消费者.假设消费者在阅读第一条记录后离线一段时间.当消费者宕机时，日志压缩开始.如果日志压缩会删除墓碑记录，消费者将永远不会知道记录已被删除的事实.如果消费者实现了缓存，则记录可能永远不会被删除.这样，tombstone 的保存时间更长，允许离线消费者接收所有 tombstone 进行本地清理.

Tombstone records are preserved longer by design. The reason is, that brokers don't track consumers. Assume, that a consumer goes offline for some time after reading the first record. While the consumer is down, log compaction kicks. If log compaction would delete the tombstone record, the consumer would never learn about the fact, that the record was deleted. If the consumer implements a cache, it could happen that the record gets never deleted. Thus, tombstone are preserved longer to allow offline consumer to receive all tombstones for local cleanup.

只有在delete.retention.ms(默认值为1天)后才会删除Tombstone.注意:这是一个主题级别的配置，没有代理级别的配置.因此，如果要更改它，则需要为每个主题设置配置.

Tombstone will be deleted only after delete.retention.ms (default value is 1 day). Note: this is a topic level configuration and there is no broker level configuration for it. Thus, you need to set the config per topic if you want to change it.

这篇关于卡夫卡不删除带有墓碑的密钥的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！