如何让消费者从 Kafka 请求超过 1MB 的记录 [英] How can I make consumer request more than 1MB records from Kafka

问题描述

每当我的消费者从 Kafka 请求一个新批次时，它总是请求 1MB 的数据，然后它似乎请求下一个 1MB，依此类推.有没有人知道接收20MB的批次需要什么配置和编程步骤?

解决方案

您可以将消费者属性中的属性 max.partition.fetch.bytes 设置为您想要的值(默认为 1MB).

此外，此值必须等于或大于代理配置中的 max.message.size 属性，以确保您的消费者能够读取代理接受的所有消息.

最后，如果处理 20MB 的时间太长，您可能需要增加消费者的 session.timeout.ms 设置(默认为 3 秒)，以避免代理认为您的消费者已死并触发重新平衡.

Whenever my consumer requests a new batch from Kafka, it is requesting always 1MB of data, then it seems to request the next 1MB, and so forth. Does anybody know what the configuration and programming steps are to to receive batches of 20MB?

解决方案

You can set the property max.partition.fetch.bytes in the consumer properties to the value you desire (default is 1MB).

Also, this value must be equal or greater than max.message.size property in the broker configuration to be sure that your consumers will be able to read all messages accepted by the broker.

Finally, if processing 20MB takes too long, you may want to increase your session.timeout.ms setting at the consumer (defaults to 3 seconds), to avoid the broker thinking your consumer is dead and triggering a rebalance.

这篇关于如何让消费者从 Kafka 请求超过 1MB 的记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！