Kafka无法跟踪上次提交的偏移量 [英] Kafka fails to keep track of last-commited offset

问题描述

kakfa-broker在管理偏移量方面是否存在任何已知问题?顺便说一句，我们面临的问题是当我们尝试重新启动kafka-consumer(即应用程序重新启动)时，有时所有偏移量都重置为0.完全不知道为什么消费者不能从最后提交的偏移量开始.

Is there any known issue with kakfa-broker in managing the offsets? Bcz, problem which we are facing is when we try to restart of kafka-consumer(i.e, app restart) sometimes all the offset are reset to 0. Completely clueless on why are consumers not able to start from the last commited offset.

我们最终会在产品中面临这个问题，其中整个q事件都将被重播:

We are eventually facing this issue in prod wherein the whole q events are replayed again :

• spring-boot版本-2.2.6发行版
• spring-kafka-2.3.7版本
• kafka-client -2.3.1
• apache-kafka-kafka_2.12-2.3.1

我们有10个主题，每个主题属于同一组，每个主题有50个分区，我们会在运行时根据负载增加主题分区和使用者数量.

We have 10 topics with 50 partitions for each topic which belongs to same group, we increase topic-partition and consumer count at run-time based on load.

• 自动提交=假
• 处理后同步提交每个偏移量
• 最大投票记录设置为1

在完成所有此配置后，该配置将在本地设置中按预期运行，在部署到产品后，我们发现此类问题并非每次重启时都存在.有没有我所缺少的配置.完全无能为力！！！！

After all this config it runs as expected in local setup, after deployed to prod we see such issues nut not at every restart. Is there any config that i'm missing. Completely Clueless!!!!!

推荐答案

请勿根据其他答案中的建议启用自动提交；侦听器容器将更可靠地提交偏移量，并且正如您所说的，您始终不会遇到问题.

Do not enable auto commit per the suggestion in another answer; the listener container will more reliably commit the offsets and, as you say, you don't have the problem all the time.

是否有可能一周没有收到任何记录?

Is it possible that you receive no records for a week?

或者，您的经纪人是否可能具有较短的 offsets.retention.minutes 属性?

Or, is it possible that your broker has a shorter offsets.retention.minutes property?

在2.0中，它从默认的1天更改为1周.如果偏移量已过期而被删除，而您重新启动使用者，则会得到所观察到的行为.

In 2.0, it was changed from a 1 day default to 1 week. If the offsets have been removed because they expired and you restart the consumer, you'll get the behavior you observe.

这篇关于Kafka无法跟踪上次提交的偏移量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！