即使设置 acks=all，Kafka 生产者也会丢失消息 [英] Kafka producer losing message even if set acks=all

问题描述

这是我的配置:

props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, "1");
props.put(ProducerConfig.LINGER_MS_CONFIG, "1");

和

try {
    producer.send(record);
} catch (Throwable ex) {
    log.error(ex, "exception.");
}

但我们发现消息丢失.

网络抖动是否会导致这种情况?

if the network dithering will lead this?

我们需要用回调发送吗?

do we need to send with Callback?

producer.send(record, new Callback() {
  @Override
  public void onCompletion(RecordMetadata metadata, Exception exception) {}
})

推荐答案

Kafka Producer 基本上有三种不同的模式来向 Kafka 生产消息:

A Kafka Producer has basically three different modes to produce messages to Kafka:

• 即发即忘
• 同步
• 异步

当仅调用 producer.send(record) 时，您正在应用 fire-and-forget 模式.配置 acks 仅确保如果所有复制都确认了该消息，则该消息将被视为成功写入 Kafka.但是，您的制作人不会等待回复.

When only calling producer.send(record) you are applying the fire-and-forget mode. The configuration acks only ensures that the message will be seen as a successful write to Kafka if all replications have acknowledged that message. However, your producer does not wait for the reply.

正如您所提到的，您可以使用回调来了解代理的回复.这将是异步模式.但是不要忘记还要flush()缓冲的记录.

As you have mentioned, you could make use of a callback to understand the reply from the broker. This will be the asychronous mode. But do not forget to also flush() the buffered records.

另一种选择是应用同步，这可以通过阻塞方法 get 实现，该方法等待来自代理的 ack 响应.你只需要把唯一的一行代码改成

Another option would be to apply the synchronous which can be achieved by a blocking method get that waits for the ack-response from the broker. You only have to change the only line of code to

producer.send(record).get();

将重试次数增加到大于 1 可能是值得的.Producer Callback 可以返回可重试 异常，这些异常可以通过多次重试来解决.要了解所有回调异常，您可以查看关于 Producer Callback.

It might be worth increasing the retries to a number larger than 1. The Producer Callback can return retrieable exceptions which could be solved with more than one retry. To understand all callback exceptions you can have a look at another SO question on Producer Callback.

这篇关于即使设置 acks=all，Kafka 生产者也会丢失消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！