librdkafka生产者如何了解Kafka中的新主题分区 [英] How does librdkafka producer learn about new topic partitions in Kafka
问题描述
我正在运行 rdkafka_simple_producer.c 来向 Kafka 集群生成消息.我有一个主题和 30 个分区.使用默认的循环分区器.当生产者正在工作并向 Kafka 生成消息时,我向 Kafka 添加了更多分区
I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to Kafka
kafka/bin/kafka-topics.sh --alter --zookeeper server2:2181 --topic demotest --partitions 40
我希望制作人能够意识到这一变化,并最终开始制作所有 40 个主题.但是,最后我只看到数据生成到原来的 30 个分区.
I'd expect the producer to realize about the change and eventually begin to produce to all 40 topics. However, at the end I only see data was produced to the original 30 partitions.
在测试中,生产者运行了 2 分钟.
In the test the producer ran for 2 minutes.
我需要在 simple_producer 中添加任何函数调用还是我需要考虑的 Kafka 参数?
Do I need to add any function call in the simple_producer or is it a Kafka parameter I need to consider?
提前致谢!
推荐答案
这个问题我终于有了答案.实际上,生产者会定期刷新有关元数据的信息.间隔由配置参数topic.metadata.refresh.interval.ms定义
I finally have a response for this question. Actually the producer is refreshed periodically about metadata. The interval is defined by configuration parameter topic.metadata.refresh.interval.ms
topic.metadata.refresh.interval.ms 的默认值为 300000(以毫秒为单位).这是 5 分钟,测试只持续了 2 分钟.生产者将在元数据刷新后自动识别新添加的分区.
The value of topic.metadata.refresh.interval.ms is 300000 (in ms) by default. This is 5 minutes and test lasted only 2 minutes. The producer will recognize the new added partitions automatically after the metadata refresh.
因此,对于遇到相同问题的任何人,您都可以配置此参数以加快元数据刷新.无需编码,只需通过以下方式设置此值:
So, for anybody with the same problem, you can configure this parameter to have faster metadata refresh. No coding needed more than setting this value with:
rd_kafka_conf_set(conf, "topic.metadata.refresh.interval.ms", "15000",
errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
当然,您需要考虑这意味着代理和生产者之间有更多的消息传递.更多关于 librdkafka 配置:librdkafka 配置
Of course, you need to consider this implies more messaging between a broker and a producer. More on librdkafka configuration: librdkafka configuration
这篇关于librdkafka生产者如何了解Kafka中的新主题分区的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!