Spring Boot Kafka:由于组已经重新平衡，因此无法完成提交 [英] Spring Boot Kafka: Commit cannot be completed since the group has already rebalanced

问题描述

今天，在我的 Spring Boot 和单实例 Kafka 应用程序中，我遇到了以下问题:

Today, in my Spring Boot and single instance Kafka application I faced the following issue:

org.apache.kafka.clients.consumer.CommitFailedException:提交不能完成，因为该组已经重新平衡并分配了分区给另一个成员.这意味着之间的时间对 poll() 的后续调用比配置的要长max.poll.interval.ms，这通常意味着轮询循环是花费太多时间处理消息.你可以解决这个问题通过增加会话超时或减少最大大小使用 max.poll.records 在 poll() 中返回的批次.

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

这可能是什么原因以及如何解决?据我了解 - 我的消费者被阻塞了很长时间并且没有响应心跳.我应该调整 Kafka 属性以解决它.你能告诉我我应该调整哪些确切的属性以及在哪里，例如在 Kafka 端还是在我的应用程序 Spring Kafka 端?

What may be the reason for this and how to fix it? As far as I understand - my consumer was blocked for a long time and didn't respond for the heartbeat. And I should adjust Kafka properties in order to address it. Could you please tell me what exact properties should I adjust and where, for example on the Kafka side or on my application Spring Kafka side?

推荐答案

默认情况下，Kafka 会返回一批记录 fetch.min.bytes(默认为 1)，最大为 max.poll.records(默认500)，或者fetch.max.bytes(默认52428800)，否则会等待fetch.wait.max.ms(默认 100)，然后再返回一批数据.您的消费者需要对该数据做一些工作，然后再次调用 poll().您的消费者的工作预计将在 max.poll.interval.ms 内完成(默认为 300000 — v2.0 之前的 5 分钟和 v2.0 之后的 30000 - 30 秒).如果在此超时到期之前没有调用 poll()，则认为消费者失败，组将重新平衡以将分区重新分配给另一个成员.

By default Kafka will return a batch of records of fetch.min.bytes (default 1) up to either max.poll.records (default 500), or fetch.max.bytes (default 52428800), otherwise it will wait fetch.wait.max.ms (default 100) before returning a batch of data. Your consumer is expected to do some work on that data and then call poll() again. Your consumer's work is expected to be completed within max.poll.interval.ms (default 300000 — 5 mins in pre v2.0 and 30000 - 30 seconds post v2.0). If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member.

因此，要解决您的问题，请减少返回的消息数量，或增加 max.poll.interval.ms 属性以避免超时和重新平衡.

So to fix your issue, reduce the number of messages returned, or increase max.poll.interval.ms property to avoid timing out and rebalancing.

这篇关于Spring Boot Kafka:由于组已经重新平衡，因此无法完成提交的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！