__consumer_offsets 具有非常大分区的主题 [英] __consumer_offsets topic with very big partitions

问题描述

我使用的是 Kafka 2.0.0__consumer_offsets 主题的一些分区是 500-700 GB 和超过 5000-7000 个段.这些段早于 2-3 个月.日志中没有错误，并且该主题默认为 COMPACT.

I am using Kafka 2.0.0 There are some partitions of the __consumer_offsets topic that are 500-700 GB and more than 5000-7000 segments. These segments are older than 2-3 months. There aren't errors in the logs and that topic is COMPACT as default.

可能是什么问题?也许是配置或消费者问题?或者可能是 kafka 2.0.0 的错误?我可以做哪些检查?

What could be the problem? Maybe a config or a consumer problem? or maybe a bug of kafka 2.0.0? What checks could I do?

我的设置:

log.cleaner.enable=true
log.cleanup.policy = [delete]
log.retention.bytes = -1
log.segment.bytes = 268435456
log.retention.hours = 72
log.retention.check.interval.ms = 300000



offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 10080
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 3
offsets.topic.segment.bytes = 104857600

推荐答案

可能是您的应用程序每时每刻都在与不同的消费者群体循环?

Could be that you have application looping with different consumer groups evey time ?

您可以使用此命令查看您的 _consumer_offsets 主题内部，尝试找出消费者组名重复，也许你有一些用户用循环创建了许多消费者组或运行控制台消费者......

You could use this command to look inside your _consumer_offsets topic, try to find out consumer Group names repeats, maybe you have some users creating many consumer groups with loops or running console consumers ...

echo "exclude.internal.topics=false" > /tmp/consumer.config
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning

这篇关于__consumer_offsets 具有非常大分区的主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！