Kafka 控制台消费者错误“分区上的偏移提交失败" [英] Kafka console consumer ERROR "Offset commit failed on partition"

问题描述

我正在使用 kafka-console-consumer 来探索 kafka 主题.

I am using a kafka-console-consumer to probe a kafka topic.

间歇性地，我收到此错误消息，然后是 2 个警告:

Intermittently, I am getting this error message, followed by 2 warnings:

[2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed on partition my-topic-0 at offset 444: The coordinator is not aware of this member. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Asynchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=444, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Synchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=447, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

它在警告日志中建议:

这意味着对 poll() 的后续调用之间的时间比配置的 max.poll.interval.ms 长，这通常意味着轮询循环在消息处理上花费了太多时间.您可以通过增加会话超时或通过使用 max.poll.records 减少 poll() 中返回的批次的最大大小来解决此问题.

This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

所以，我要么增加max.poll.interval.ms，要么减少max.poll.records.

So, I either need to increase max.poll.interval.ms or decrease max.poll.records.

请告知每种方法的含义，以及在不同情况下首选哪种方法?

Please advise what would be the implication of each method, and which one is preferred on a different situation?

推荐答案

如果你增加 max.poll.interval.ms 表示花时间处理大量记录是可以的"和如果您能比小批量更有效地处理大批量，您将获得吞吐量.

If you increase max.poll.interval.ms that says "it’s ok to spend time processing a large batch of records" and you’ll gain throughput if you can process larger batches more efficiently than smaller ones.

减少 max.poll.records 表示减少记录，以便有足够的时间来处理它们"，并且有利于延迟而不是吞吐量.

To decrease max.poll.records says "take fewer records so there’s enough time to process them" and would favor latency over throughput.

还要考虑到两者都配置得很好，但是其他原因导致了 poll 循环中的性能问题.我会在更改配置之前先探索一下，以免掩盖更大的问题.

Also consider that both are configured fine, but something else is causing performance issues within your poll loop. I would explore that first before changing configuration so you don’t mask a bigger problem.

这篇关于Kafka 控制台消费者错误“分区上的偏移提交失败"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！