如何增加每批Spring Kafka Consumer消耗的消息数? [英] How to increase the number of messages consumed by Spring Kafka Consumer in each batch?

问题描述

我正在构建一个Kafka Consumer应用程序，该应用程序使用来自Kafka Topic的消息并执行数据库更新任务.每天都会大批量生成消息-因此，该主题在10分钟内加载了大约100万条消息.主题有8个分区.

I am building a Kafka Consumer application that consumes messages from a Kafka Topic and performs a database update task. The messages are produced in a large batch once every day - so the Topic has about 1 million messages loaded in 10 minutes. The Topic has 8 partitions.

Spring Kafka使用者(用@KafkaListener注释并使用ConcurrentKafkaListenerContainerFactory批注)在非常短的时间内触发.

The Spring Kafka Consumer (annotated with @KafkaListener and using a ConcurrentKafkaListenerContainerFactory) is triggered in very short batches.

批处理大小有时仅为1或2条消息.如果它一次可以消耗约1000条消息并一起处理(例如，我可以在一个更新SQL中更新数据库)，而不是为每条消息连接到数据库，那么这将对性能有所帮助.

The batch size is sometimes just 1 or 2 messages. It would help performance if it can consume about 1000 messages at once and process it together (for example, I could update the database in a single update SQL), instead of connecting to the database for each message.

我已经尝试在工厂中减少并发，以避免多个线程消耗较少的消息.

I have already tried to decrease the concurrency in the factory to avoid multiple threads consuming smaller number of messages.

我还将Kafka的server.properties中的 socket.send.buffer.bytes 属性从102400增加到1024000.

I also increased the socket.send.buffer.bytes property in Kafka's server.properties to 1024000, from 102400.

这些步骤并未增加批量大小.

These steps have not increased the batch size.

还有其他配置可以用来增加消费者的浴室大小吗?

Is there any other configuration I could use to increase the bath size of the consumer?

推荐答案

请参阅kafka用户属性 max.poll.records ， fetch.min.bytes ，fetch.max.wait.ms ， fetch.max.bytes ， max.partition.fetch.bytes .

See kafka consumer properties max.poll.records, fetch.min.bytes, fetch.max.wait.ms, fetch.max.bytes, max.partition.fetch.bytes.

您最需要的 fetch.min.bytes 和 fetch.max.wait.ms .

这篇关于如何增加每批Spring Kafka Consumer消耗的消息数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！