Kafka控制台使用者错误“分区上的偏移提交失败" [英] Kafka console consumer ERROR "Offset commit failed on partition"

问题描述

我正在使用kafka-console-consumer来探讨kafka主题.

I am using a kafka-console-consumer to probe a kafka topic.

间歇性地，我收到此错误消息，后面是2条警告:

Intermittently, I am getting this error message, followed by 2 warnings:

[2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed on partition my-topic-0 at offset 444: The coordinator is not aware of this member. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Asynchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=444, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Synchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=447, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

它在警告日志中建议:

这意味着后续调用poll()之间的时间比配置的max.poll.interval.ms更长，这通常意味着轮询循环在消息处理上花费了太多时间.您可以通过增加会话超时或通过使用max.poll.records减小poll()中返回的批处理的最大大小来解决此问题.

This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

因此，我需要增加max.poll.interval.ms或减少max.poll.records.

So, I either need to increase max.poll.interval.ms or decrease max.poll.records.

请告知每种方法的含义，以及在不同情况下首选哪种方法?

Please advise what would be the implication of each method, and which one is preferred on a different situation?

推荐答案

如果增加max.poll.interval.ms则表示可以花大量时间处理大量记录"，并且如果可以处理较大的批次，则可以提高吞吐量.比较小的产品更有效.

If you increase max.poll.interval.ms that says "it’s ok to spend time processing a large batch of records" and you’ll gain throughput if you can process larger batches more efficiently than smaller ones.

要减少max.poll.records表示减少记录数量，因此有足够的时间来处理它们"，并且优先考虑延迟而不是吞吐量.

To decrease max.poll.records says "take fewer records so there’s enough time to process them" and would favor latency over throughput.

还要考虑两者都配置良好，但是其他原因也会导致poll循环内的性能问题.在更改配置之前，我会先进行探讨，这样您就不会掩盖更大的问题.

Also consider that both are configured fine, but something else is causing performance issues within your poll loop. I would explore that first before changing configuration so you don’t mask a bigger problem.

这篇关于Kafka控制台使用者错误“分区上的偏移提交失败"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！