改善Kafka Producer的性能 [英] Improving performance of Kafka Producer

问题描述

我们在apache kafka 0.10.0.x和spring 3.x上运行，并且不能使用spring kafka，因为它在spring框架版本4.x中受支持.

We're running on apache kafka 0.10.0.x and spring 3.x and cannot use spring kafka as it is supported with spring framework version 4.x.

因此，我们正在使用本地的Kafka Producer API生成消息.

Therefore, we are using the native Kafka Producer API to produce messages.

现在，我所担心的是我的制片人的表现.事实是，我相信对 producer.send 的调用才真正建立了与Kafka代理的连接，然后将消息放入缓冲区，然后尝试发送，然后可能调用您提供的回调方法在 producer.send()中.

Now the concern that i have is the performance of my producer. The thing is i believe a call to producer.send is what really makes the connection to the Kafka broker and then puts the message onto the buffer and then attempts to send and then possibly calls your the provided callback method in the producer.send().

现在，KafkaProducer文档说它使用一个缓冲区和另一个I/O线程来执行发送，因此应该适当地关闭它们，以免资源泄漏.

Now the KafkaProducer documentation says that it uses a buffer and another I/O thread to perform the send and that they should be closed appropriately so that there is no leakage of resources.

据我了解，这意味着如果我每次调用 producer.send()时都会发送100条消息，它将尝试连接到代理，这是一项昂贵的I/O操作

From what i understand, this means that if i have 100s of messages being sent every time i invoke producer.send() it attempts to connect to the broker which is an expensive I/O operation.

如果我错了，或者我建议使用更好的KafkaProducer，可以请您纠正我的理解吗?

Can you please correct my understanding if i am wrong or maybe suggest a better to use the KafkaProducer?

推荐答案

您的理解部分正确.

@leshkin指出，有一些配置参数可以调整 KafkaProducer 如何处理要发送的邮件的缓冲.

As @leshkin pointed out there are configuration parameters to tune how the KafkaProducer will handle buffering of messages to be sent.

但是，独立于缓冲策略之外，制作人将负责缓存与主题领导者经纪人之间已建立的连接.

However independently from the buffering strategy, the producer will take care of caching established connections to topic-leader brokers.

实际上，您可以使用 connections.max.idle.ms 参数(默认值为9分钟)来调整生产者将这种连接保持多长时间.

Indeed you can tune for how long the producer will keep such connection around using the connections.max.idle.ms parameter (defaults to 9 minutes).

因此，为了回答您的原始问题，与代理建立连接的I/O成本仅在第一次 send 调用时发生，并且会随着时间的流逝而摊销.发送.

So to respond to your original question, the I/O cost of establishing a connection to the broker happens only on the first send invocation and will be amortised over time as long as you have data to send.

这篇关于改善Kafka Producer的性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！