kafka异步发送中确认模式有什么影响? [英] What is the impact of acknowledgement modes in asynchronous send in kafka?

问题描述

当我们使用异步 send() 调用发送消息时，确认模式 ('0','1','all') 有什么影响吗?

Does acknowledgement modes ('0','1','all') have any impact, when we send messages using asynchronous send() call?

我尝试测量发送调用延迟(即通过记录调用 send() 方法前后的时间)并观察到两者(异步发送，acks=1) 和(异步发送，acks=all)花费相同的时间.

I have tried measuring the send call latency (that is, by recording time before and after call to send() method) and observed that both (asynchronous send, acks=1) and (asynchronous send, acks=all) took same time.

但是，吞吐量数字存在明显差异.

However, there is a clear difference in throughput numbers.

producer.send(record, new ProducerCallback(record));

我认为即使在异步模式下使用 acks=all 时，发送调用也会被阻止.有人能解释一下确认模式 ('0','1','all') 在异步模式下是如何工作的吗?

I thought send call will be blocked when we use acks=all even in asynchronous mode. Can someone explain how acknowledgement modes ('0','1','all') work in asynchronous mode?

推荐答案

根据文档:

公共未来发送(ProducerRecord记录，回调回调)

public Future send(ProducerRecord record, Callback callback)

异步发送一条记录到一个主题并调用提供的当发送被确认时回调.发送是异步并且这个方法会在记录完成后立即返回存储在等待发送的记录缓冲区中.这允许并行发送许多记录而不会阻塞等待一一回复.

Asynchronously send a record to a topic and invoke the provided callback when the send has been acknowledged. The send is asynchronous and this method will return immediately once the record has been stored in the buffer of records waiting to be sent. This allows sending many records in parallel without blocking to wait for the response after each one.

因此，有一点可以肯定，异步发送"并不真正关心acks"配置是什么.它所做的只是将消息推送到缓冲区中.一旦这个 buffer 开始被处理(由 linger.ms 和 batch.size 属性控制)，就会检查acks".

So, one thing is certain that the asynchronous "send" does not really care about what the "acks" config is. All it does is push the message into a buffer. Once this buffer starts getting processed (controlled by linger.ms and batch.size properties), then "acks" is checked.

如果 acks=0 -> 只是开火然后忘记.Producer 不会等待确认.

If acks=0 -> Just fire and forget. Producer won't wait for an acknowledgement.

如果 acks=1-> 当消息成功写入领导者时，代理发送确认.

If acks=1-> Acknowledgement is sent by the broker when message is successfully written on the leader.

如果 acks=all -> 当消息成功写入所有副本时，代理发送确认.

If acks=all -> Acknowledgement is sent by the broker when message is successfully written on all replicas.

在 1 和 all 的情况下，这将成为阻塞调用，因为生产者将等待确认，但您可能不会注意到这一点，因为它发生在并行线程上.在 acks=all 的情况下，预计 ack 到达所需的时间会比 acks=1 长一点(网络延迟和副本数量是显而易见的原因).

In case of 1 and all, this becomes a blocking call as the producer will wait for an acknowledgement but, you might not notice this as it happens on a parallel thread. In case of acks=all, it is expected that it will take a little longer for the ack to arrive than acks=1 (network latency and number of replicas being the obvious reasons).

此外，您应该在异步生产者中配置重试"属性，以便如果确认失败(由于任何原因，例如数据包损坏/丢失)，生产者知道它应该再次尝试发送消息的次数(增加交货保证).

Further, you should configure "retries" property in your async-producer so that, if an acknowledgement fails (due to any reason e.g. packet corrupted/lost), the producer knows how many times it should try again to send the message (increase the guarantee of delivery).

最后:但是，吞吐量数字存在明显差异."-- 这是因为从代理到生产者线程的确认延迟.

Lastly: "However, there is a clear difference in throughput numbers." -- That's true because of the acknowledgement latency from the broker to the producer thread.

希望有帮助！:)

这篇关于kafka异步发送中确认模式有什么影响?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！