清理策略:压缩/删除并保留日志 [英] Cleanup Policy:Compact/Delete and log.retention
问题描述
我对Kafka Topic清理政策及其与log.retention ...的相互作用有疑问.
I have a question about Kafka Topic cleanup policies and their interaction of log.retention....
例如，如果我将cleanup.policy设置为compact，那么压缩将仅在主题的保留时间之后开始，或者保留时间对压缩没有影响?
For example, if I set cleanup.policy to compact, compaction will only start after the retention time of the topic or retention time has no effect for compaction?
问题的第二部分，如果我一起使用压缩，则删除，并且我有log.retention可以说1天，话题一直压缩，但是话题的内容会在一天后被删除吗?还是一天后实现压缩和删除?
Second part of the question, if I use compact,delete together, and I have log.retention for lets say 1 day, topic compacted all the time but content of the topic will be deleted after one day? or compaction and delete realised after one day?
寻求答案...
推荐答案
可以删除或压缩日志段或同时压缩日志段以管理其大小.主题级别的配置 cleanup.policy 确定管理主题日志段的方式.
Log segments can be deleted or compacted, or both, to manage their size. The topic-level configuration cleanup.policy determines the way the log segments for the topic are managed.
通过压缩清除日志
如果主题级别的配置 cleanup.policy 设置为 compact ，则日志清理器会在后台定期压缩主题日志.
If the topic-level configuration cleanup.policy is set to compact,the log for the topic is compacted periodically in the background by the log cleaner.
在一个紧凑的主题中，日志仅需要包含每个键的最新消息，而较早的消息可以被丢弃.
In a compacted topic,the log only needs to contain the most recent message for each key while earlier messages can be discarded.
无需将log.retention设置为-1或任何其他值.您的主题将被压缩，并且旧消息绝不会被删除(按照压缩规则).
There is no need to set log.retention to -1 or any other value. Your topics will be compacted and old messages never deleted (as per compaction rules).
请注意，只有非活动文件段可以压缩；活动段永远不会被压缩.
Note that only the inactive file segment can be compacted; active segment will never be compacted.
同时使用两者进行日志清除
您可以同时为 cleanup.policy 配置指定 delete 和 compact 值.在这种情况下，将压缩日志，但是清除过程还将遵循保留时间或大小限制设置.
You can specify both delete and compact values for the cleanup.policy configuration at the same time. In this case, the log is compacted, but the cleanup process also follows the retention time or size limit settings.
我建议您通过以下链接
https://ibm.github.io/event-streams/installing/capacity-planning/
