Kafka - 日志压缩行为 [英] Kafka - Log compaction behavior

问题描述

我正在尝试理解日志压缩.

I am trying to understand the log compaction.

对于一个主题，我的设置是

For a topic, my settings are

min.cleanable.dirty.ratio   0.005
segment.ms  5000
cleanup.policy  compact

我为该主题制作了以下消息

I produced below messages to the topic

a   0
b   1
c   1
f   4
r   0
s   1
u   1
f   5
h   1

我正在尝试通过 kafka 控制台消费者进行消费者(即使在等待 5 分钟并创建了一个新的控制台消费者之后) - 我仍然看到 2 次 f

I am trying to consumer via kafka console consumer (even after waiting 5 mins and created a new console consumer) - i still see the 2 occurrences of f

a   0
b   1
c   1
f   4
r   0
s   1
u   1
f   5
h   1

不应该删除吗?

推荐答案

在压缩过程中只能考虑不在活动段中的消息.即使你设置了segment.ms=5000，当出现分区的新消息时，也可以滚动新的日志段.

Only messages that are not in active segment can be consider in compaction process. Even if you set segment.ms= 5000, rolling new log segment can be made when new messages for partition appear.

如果您一次发送所有消息，则所有消息都将发送到同一段，并且该段将处于活动状态.如果新消息将在 5000 毫秒后到达，则它对压缩有效.log.retention.check.interval.ms 之后的日志清理器将尝试压缩具有相同密钥的消息.

If you send all message at once, all messages will go to the same segment and that segment will be active. It became valid for compaction if new messages will arrive after 5000ms. Log cleaner after log.retention.check.interval.ms will try to compact messages with same key.

这篇关于Kafka - 日志压缩行为的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！