librdkafka生产者如何了解Kafka中的新主题分区 [英] How does librdkafka producer learn about new topic partitions in Kafka

问题描述

我正在运行rdkafka_simple_producer.c来向Kafka集群生成消息.我有一个主题和30个分区.使用默认的循环分区程序. 当生产者正在工作并向Kafka生成消息时，我向Kafka添加了更多分区

I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to Kafka

kafka/bin/kafka-topics.sh --alter --zookeeper server2:2181 --topic demotest --partitions 40

我希望制片人意识到这一变化，并最终开始制作所有40个主题. 但是，最后，我只看到数据生成到最初的30个分区.

I'd expect the producer to realize about the change and eventually begin to produce to all 40 topics. However, at the end I only see data was produced to the original 30 partitions.

在测试中，制作人跑了2分钟.

In the test the producer ran for 2 minutes.

我是否需要在simple_producer中添加任何函数调用，还是我需要考虑的Kafka参数?

Do I need to add any function call in the simple_producer or is it a Kafka parameter I need to consider?

提前谢谢！

推荐答案

对于这个问题，我终于得到了答复.实际上，生产者会定期刷新有关元数据的信息.该间隔由配置参数topic.metadata.refresh.interval.ms

I finally have a response for this question. Actually the producer is refreshed periodically about metadata. The interval is defined by configuration parameter topic.metadata.refresh.interval.ms

topic.metadata.refresh.interval.ms的值默认为300000(以ms为单位).这是5分钟，测试仅持续2分钟. 生产者将在元数据刷新后自动识别新添加的分区.

The value of topic.metadata.refresh.interval.ms is 300000 (in ms) by default. This is 5 minutes and test lasted only 2 minutes. The producer will recognize the new added partitions automatically after the metadata refresh.

因此，对于有相同问题的任何人，您都可以配置此参数以更快地刷新元数据.除了通过以下方式设置该值，无需编码:

So, for anybody with the same problem, you can configure this parameter to have faster metadata refresh. No coding needed more than setting this value with:

rd_kafka_conf_set(conf, "topic.metadata.refresh.interval.ms", "15000",
                          errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)

当然，您需要考虑到这意味着经纪人和生产者之间会有更多的消息传递. 有关librdkafka配置的更多信息: librdkafka配置

Of course, you need to consider this implies more messaging between a broker and a producer. More on librdkafka configuration: librdkafka configuration

这篇关于librdkafka生产者如何了解Kafka中的新主题分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！