卡夫卡生产者即使设置了acks = all也丢失了消息 [英] Kafka producer losing message even if set acks=all

问题描述

这是我的配置:

props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, "1");
props.put(ProducerConfig.LINGER_MS_CONFIG, "1");

和

try {
    producer.send(record);
} catch (Throwable ex) {
    log.error(ex, "exception.");
}

但是我们发现消息丢失.

but we found message missing.

如果网络抖动会导致这种情况?

if the network dithering will lead this?

我们需要发送回叫吗?

producer.send(record, new Callback() {
  @Override
  public void onCompletion(RecordMetadata metadata, Exception exception) {}
})

推荐答案

Kafka Producer基本上具有三种不同的模式来向Kafka生成消息:

A Kafka Producer has basically three different modes to produce messages to Kafka:

• 一劳永逸
• 同步
• 异步

仅调用producer.send(record)时，您将应用即发即弃模式.配置acks仅确保如果所有复制均已确认该消息，则该消息将被视为对Kafka的成功写入.但是，您的生产者不等待答复.

When only calling producer.send(record) you are applying the fire-and-forget mode. The configuration acks only ensures that the message will be seen as a successful write to Kafka if all replications have acknowledged that message. However, your producer does not wait for the reply.

如前所述，您可以利用回调来了解经纪人的答复.这将是异步模式.但是不要忘记也flush()缓冲的记录.

As you have mentioned, you could make use of a callback to understand the reply from the broker. This will be the asychronous mode. But do not forget to also flush() the buffered records.

另一种选择是应用 synchronous ，这可以通过等待代理发出的确认响应的阻塞方法get来实现.您只需将唯一的代码行更改为

Another option would be to apply the synchronous which can be achieved by a blocking method get that waits for the ack-response from the broker. You only have to change the only line of code to

producer.send(record).get();

可能值得将重试次数增加到大于1的次数.生产者Callback可以返回可检索的异常，可以通过多次重试来解决该异常.要了解所有回调异常，您可以在生产者回调.

It might be worth increasing the retries to a number larger than 1. The Producer Callback can return retrieable exceptions which could be solved with more than one retry. To understand all callback exceptions you can have a look at another SO question on Producer Callback.

这篇关于卡夫卡生产者即使设置了acks = all也丢失了消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！