kafka 旧的 consumer_offsets 没有被删除 [英] kafka Old consumer_offsets are not getting deleted

问题描述

Kafka 版本 0.10.1.1

经纪人拥有所有日志清理/保留默认值:

The brokers are having all the log cleanup/retention defaults:

log.cleanup.policy=delete
log.retention.hours=168 (7 days)
offsets.retention.minutes=1440 (1 day)
log.index.size.max.bytes=10485760 (10M)

但我可以看到 7 月 6 日的 __consumer_offsets:

But I can see the __consumer_offsets from Jul 6:

[root@npe-tfb-kfk-w2b-b3 __consumer_offsets-13]# ls -ltr -h
total 71M
-rw-r-----. 1 root root 70M Jul  6 08:31 00000000000081206436.log
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.index
-rw-r-----. 1 root root 10M Jul 26 12:10 00000000000081206436.timeindex

在生产中，我可以看到 7 月 16 日的 consumer_offsets .log 文件.

In Production I can see consumer_offsets .log file from Jul 16.

这怎么可能?如果我手动删除旧的.log、.index 和.timeindex 文件，是否会影响集群的稳定性.

How is this possible? If I manually delete the old .log, .index and .timeindex files, is that going to impact the stability of the cluster.

推荐答案

这里的日期差异很可能是由于重新启动和非正常关机造成的.在这种情况下重建索引文件.就您的偏移主题而言，除非您确定这就是您想要的，否则您不应手动删除它.它可能会导致您的消费者失去状态.该主题已压缩，您根本没有太多数据.我希望您之前已经删除了段，因为我在那里没有看到日志段 0.只要确保您的日志清理器仍然正常工作，您就会没事的.如果您因为 70MB 的日志段而担心磁盘空间，那么您需要更多的磁盘.Kafka 磁盘的大小应为 100 GB.

The date discrepancy here is likely because of a restart with an unclean shutdown. The index files are rebuilt on such cases. As far as your offsets topic goes, you shouldn't manually deleted it unless you know for sure that's what you want. It will potentially cause a loss of state for your consumers. The topic is compacted and you don't have much data in it at all. I expect that you have previously deleted segments before because I don't see log segment 0 there. Just make sure your log cleaner is still functional and you will be ok. If you are worried about disk space because of a 70MB log segment, then you need more disk. Kafka disks should be 100s of GB in size.

这篇关于kafka 旧的 consumer_offsets 没有被删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！