Spring Boot Kafka:无法完成提交，因为该组已经重新平衡 [英] Spring Boot Kafka: Commit cannot be completed since the group has already rebalanced

问题描述

今天，在我的Spring Boot和单实例Kafka应用程序中，我遇到了以下问题:

Today, in my Spring Boot and single instance Kafka application I faced the following issue:

org.apache.kafka.clients.consumer.CommitFailedException:提交无法 由于该组已经重新平衡并分配了 分区到另一个成员.这意味着 后续对poll()的调用的时间比配置的时间长 max.poll.interval.ms，通常表示轮询循环为 花太多时间处理邮件.您可以解决这个问题 通过增加会话超时或减小最大大小 在poll()中返回的批次具有max.poll.records.

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

这可能是什么原因以及如何解决?据我了解-我的消费者很长一段时间都被封锁，对心跳没有反应.我应该调整Kafka属性以解决该问题.您能否告诉我应该调整哪些确切属性，例如在Kafka端或我的应用程序Spring Kafka端?

What may be the reason for this and how to fix it? As far as I understand - my consumer was blocked for a long time and didn't respond for the heartbeat. And I should adjust Kafka properties in order to address it. Could you please tell me what exact properties should I adjust and where, for example on the Kafka side or on my application Spring Kafka side?

推荐答案

默认情况下，Kafka将返回一批fetch.min.bytes(默认为1)直到max.poll.records(默认为500)或fetch.max.bytes(默认值52428800)，否则它将在返回一批数据之前等待fetch.wait.max.ms(默认值100).希望您的使用者对这些数据进行一些处理，然后再次调用poll().您的消费者的工作有望在max.poll.interval.ms之内完成(在v2.0之前的默认值为300000-5分钟，在v2.0之后的默认值为30000-30秒).如果在此超时到期前未调用poll()，则认为使用方失败，该组将重新平衡以将分区重新分配给另一个成员.

By default Kafka will return a batch of records of fetch.min.bytes (default 1) up to either max.poll.records (default 500), or fetch.max.bytes (default 52428800), otherwise it will wait fetch.wait.max.ms (default 100) before returning a batch of data. Your consumer is expected to do some work on that data and then call poll() again. Your consumer's work is expected to be completed within max.poll.interval.ms (default 300000 — 5 mins in pre v2.0 and 30000 - 30 seconds post v2.0). If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member.

因此，要解决您的问题，减少返回的消息数量，或增加max.poll.interval.ms属性，以避免超时和重新平衡.

So to fix your issue, reduce the number of messages returned, or increase max.poll.interval.ms property to avoid timing out and rebalancing.

这篇关于Spring Boot Kafka:无法完成提交，因为该组已经重新平衡的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！