提高 Kafka Producer 的性能 [英] Improving performance of Kafka Producer

问题描述

我们在 apache kafka 0.10.0.x 和 spring 3.x 上运行，无法使用 spring kafka，因为 spring 框架版本 4.x 支持它.

We're running on apache kafka 0.10.0.x and spring 3.x and cannot use spring kafka as it is supported with spring framework version 4.x.

因此，我们使用原生的 Kafka Producer API 来生成消息.

Therefore, we are using the native Kafka Producer API to produce messages.

现在我担心的是我的制片人的表现.事情是我相信对 producer.send 的调用是真正建立与 Kafka 代理的连接，然后将消息放入缓冲区，然后尝试发送，然后可能调用您提供的回调方法在 producer.send().

Now the concern that i have is the performance of my producer. The thing is i believe a call to producer.send is what really makes the connection to the Kafka broker and then puts the message onto the buffer and then attempts to send and then possibly calls your the provided callback method in the producer.send().

现在 KafkaProducer 文档说它使用一个缓冲区和另一个 I/O 线程来执行发送，并且应该适当地关闭它们，以免资源泄漏.

Now the KafkaProducer documentation says that it uses a buffer and another I/O thread to perform the send and that they should be closed appropriately so that there is no leakage of resources.

据我所知，这意味着如果我每次调用 producer.send() 时都会发送 100 条消息，它会尝试连接到代理，这是一项昂贵的 I/O 操作.

From what i understand, this means that if i have 100s of messages being sent every time i invoke producer.send() it attempts to connect to the broker which is an expensive I/O operation.

如果我错了，请您纠正我的理解，或者建议更好地使用 KafkaProducer?

Can you please correct my understanding if i am wrong or maybe suggest a better to use the KafkaProducer?

推荐答案

你的理解部分正确.

正如@leshkin 指出的，有一些配置参数可以调整 KafkaProducer 将如何处理要发送的消息的缓冲.

As @leshkin pointed out there are configuration parameters to tune how the KafkaProducer will handle buffering of messages to be sent.

然而，独立于缓冲策略，生产者将负责缓存与主题领导者代理的已建立连接.

However independently from the buffering strategy, the producer will take care of caching established connections to topic-leader brokers.

实际上，您可以使用 connections.max.idle.ms 参数(默认为 9 分钟)调整生产者保持这种连接的时间.

Indeed you can tune for how long the producer will keep such connection around using the connections.max.idle.ms parameter (defaults to 9 minutes).

因此，为了回答您最初的问题，建立与代理的连接的 I/O 成本仅发生在第一次 send 调用时，并且只要您有数据可分摊发送.

So to respond to your original question, the I/O cost of establishing a connection to the broker happens only on the first send invocation and will be amortised over time as long as you have data to send.

这篇关于提高 Kafka Producer 的性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！