kafka老consumer_offsets不会被删除 [英] kafka Old consumer_offsets are not getting deleted

问题描述

Kafka版本0.10.1.1

代理具有所有日志清除/保留默认值:

The brokers are having all the log cleanup/retention defaults:

log.cleanup.policy=delete
log.retention.hours=168 (7 days)
offsets.retention.minutes=1440 (1 day)
log.index.size.max.bytes=10485760 (10M)

但是我可以从7月6日看到 __ consumer_offsets :

But I can see the __consumer_offsets from Jul 6:

[root@npe-tfb-kfk-w2b-b3 __consumer_offsets-13]# ls -ltr -h
total 71M
-rw-r-----. 1 root root 70M Jul  6 08:31 00000000000081206436.log
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.index
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.timeindex

在生产中，我可以看到7月16日起的 consumer_offsets .log 文件.

In Production I can see consumer_offsets .log file from Jul 16.

这怎么可能?如果我手动删除旧的 .log ， .index 和 .timeindex 文件，那将会影响群集的稳定性.

How is this possible? If I manually delete the old .log, .index and .timeindex files, is that going to impact the stability of the cluster.

推荐答案

此处的日期差异可能是由于重新启动而造成的关机异常.在这种情况下将重建索引文件.就偏移量主题而言，除非您确定确实要这样做，否则不应手动删除它.这可能会给您的消费者造成状态损失.该主题已压缩，您根本没有太多数据.我希望您之前已经删除了段，因为在那里我看不到日志段0.只要确保您的日志清理器仍可正常运行，您就可以了.如果由于70MB日志段而担心磁盘空间不足，则需要更多磁盘.Kafka磁盘的大小应为100 GB.

The date discrepancy here is likely because of a restart with an unclean shutdown. The index files are rebuilt on such cases. As far as your offsets topic goes, you shouldn't manually deleted it unless you know for sure that's what you want. It will potentially cause a loss of state for your consumers. The topic is compacted and you don't have much data in it at all. I expect that you have previously deleted segments before because I don't see log segment 0 there. Just make sure your log cleaner is still functional and you will be ok. If you are worried about disk space because of a 70MB log segment, then you need more disk. Kafka disks should be 100s of GB in size.

这篇关于kafka老consumer_offsets不会被删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！