卡夫卡制片人在收到第一个消息时很慢 [英] Kafka producer is slow on first message

问题描述

我在.NET中有一个测试应用程序，该应用程序创建了kafka生产者，并使用Confluent客户端库向主题发送了一些消息.

I have a test application in .NET that creates kafka producer and sends a few messages to a topic using Confluent client library.

出于某种原因，对第一条消息的确认总是迟到一秒钟，对后续消息的确认几乎立即到来.

For some reason the acknowledgement for the first message always arrives 1 second late, acknowledgements for subsequent messages arrive almost immediately.

这是正常现象还是我缺少某些配置?

• 我想我尝试调整所有生产者配置，除了设置EnableDeliveryReports = false之外没有任何帮助.
• 主题未分区，并且在发送第一条消息时已经存在.

• 如果我更新应用程序以将消息发送给几个主题，则发送给每个主题的第一条消息会延迟1秒发送.

• I think I tried tuning all producer configs, nothing helps except setting EnableDeliveryReports = false.
• Topic is not partitioned and already exists at the moment of sending the first message.

• If I update the application to send messages to a few topics, then first message to each topic gets sent with 1 second delay.

static void Main()
{
    var producer = new Producer<Null, string>(new ProducerConfig
    {
        BootstrapServers = "localhost:9092",
        LingerMs = 100,
        BatchNumMessages = 1,
    });

    for (var i = 0; i < 10; i++)
    {
        var start = DateTime.Now;
        producer.ProduceAsync(
            new TopicPartition("test-topic", Partition.Any), new Message<Null, string>
            {
                Value = $"hello kafka! #{i}"
            }).Wait(2000);

        var now = DateTime.Now;
        Console.WriteLine($"{now:HH:mm:ss.fff} Message sent in {(now - start).TotalMilliseconds:N1}.");
    }

    producer.Dispose();
    Console.ReadLine();
}

示例输出:

18:06:13.605 Message sent in 1,007.0.
18:06:13.607 Message sent in 1.0.
18:06:13.608 Message sent in 1.0.
18:06:13.609 Message sent in 1.0.
18:06:13.610 Message sent in 1.0.
18:06:13.611 Message sent in 1.0.
18:06:13.612 Message sent in 1.0.
18:06:13.613 Message sent in 1.0.
18:06:13.614 Message sent in 1.0.
18:06:13.615 Message sent in 1.0.

推荐答案

在对kafka请求日志进行更多挖掘之后，我发现在发送第一条消息之前，生产者针对所有主题发送了API_VERSIONS和METADATA请求.1秒后，它将再次发送METADATA请求，这一次是针对生产者试图将消息推送到的特定主题.

After some more digging in kafka request logs I found that before sending the first message producer sends API_VERSIONS and METADATA requests for all topics. After 1 second it sends again METADATA request, this time for the specific topic that producer is trying to push message to.

将TopicMetadataRefreshIntervalMs设置为少于1秒可以减少第一条消息的延迟.不利的一面是，生产者开始更频繁地发送元数据请求.令人惊讶的是，将TopicMetadataRefreshIntervalMs设置为超过一秒的时间不会延长延迟.

Setting TopicMetadataRefreshIntervalMs to something less than 1 second reduces the first message delay. As a downside producer starts sending metadata requests more often. Surprisingly setting TopicMetadataRefreshIntervalMs to something more than a second does not extend the delay.

不确定这是错误还是功能.

Not sure if this is a bug or a feature.

这篇关于卡夫卡制片人在收到第一个消息时很慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！