Kafka Producer 错误 TOPIC:XXXXXX 的 10 条记录过期:自批处理创建以来已经过去了 6686 毫秒加上逗留时间 [英] Kafka Producer error Expiring 10 record(s) for TOPIC:XXXXXX: 6686 ms has passed since batch creation plus linger time

问题描述

卡夫卡版本:0.10.2.1，

Kafka Version : 0.10.2.1,

Kafka Producer error Expiring 10 record(s) for TOPIC:XXXXXX: 6686 ms has passed since batch creation plus linger time
org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for TOPIC:XXXXXX: 6686 ms has passed since batch creation plus linger time

推荐答案

发生此异常是因为您排队记录的速度比发送记录的速度快得多.

This exception is occuring because you are queueing records at a much faster rate than they can be sent.

当您调用 send 方法时，ProducerRecord 将存储在内部缓冲区中以发送给代理.一旦 ProducerRecord 被缓冲，该方法立即返回，无论它是否已发送.

When you call the send method, the ProducerRecord will be stored in an internal buffer for sending to the broker. The method returns immediately once the ProducerRecord has been buffered, regardless of whether it has been sent.

记录被分组发送到代理，以减少每条消息的传输监听并提高吞吐量.

Records are grouped into batches for sending to the broker, to reduce the transport overheard per message and increase throughput.

将记录添加到批次中后，发送该批次有时间限制，以确保它已在指定的持续时间内发送.这由 Producer 配置参数 request.timeout.ms 控制，默认为 30 秒.

Once a record is added into a batch, there is a time limit for sending that batch to ensure that it has been sent within a specified duration. This is controlled by the Producer configuration parameter, request.timeout.ms, which defaults to 30 seconds.

如果批处理的排队时间超过超时限制，则会抛出异常.该批次中的记录将从发送队列中删除.

If the batch has been queued longer than the timeout limit, the exception will be thrown. Records in that batch will be removed from the send queue.

生产者配置 block.on.buffer.full、metadata.fetch.timeout.ms 和 timeout.ms 已被删除.它们最初在 Kafka 0.9.0.0 中被弃用.

因此尝试增加request.timeout.ms

不过，如果您有任何与吞吐量相关的问题，您也可以参考以下博客

Still, if you have any problem related to throughput, you can also refer following blog

这篇关于Kafka Producer 错误 TOPIC:XXXXXX 的 10 条记录过期:自批处理创建以来已经过去了 6686 毫秒加上逗留时间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！