卡夫卡vm.max_map_count [英] Kafka vm.max_map_count

问题描述

我们有一个用于Kafka流应用程序的Kafka集群.

We have a Kafka cluster for Kafka stream application.

几个小时后，我们的经纪人破产了，我们得到了 OutOfMemory 异常.

After some hours our broker went down and we got OutOfMemory exception.

我们发现 vm.max_map_count 不够，并且该进程的映射内存超过40K.

We saw the vm.max_map_count is not enough and maps memory of the process is above 40K.

有人可以解释是什么问题或对该参数有什么影响吗?

Can someone explain what can be the problem or what influence on that parameter?

数字总是增加而从不减少.

The number always increases and never goes down.

推荐答案

基于

Based on the pull request at https://github.com/apache/kafka/pull/4358/files (both the change being proposed and the comments reacting to it), it appears that each log segment (i.e. file) in each partition on each topic on the broker consumes two maps.

我希望该值会上升，直到达到稳态为止，在该状态下，所有主题的日志都已足够旧，由于保留时间间隔而开始删除.届时，每个新文件都将在删除旧文件的同时出现(假设消息速率大致恒定).我希望如果删除主题或更改现有主题或完整代理的配置(例如，减少日志保留时间或使日志翻转的频率降低)，则该值会下降；如果更改主题，则该值会下降相反的配置.

I would expect the value to rise until you reach a steady-state where all topics have logs that are old enough to start being deleted due to the retention interval. At that point, each new file would be expected to occur at around the same time as an older one is deleted (assuming roughly constant message rates). I would expect the value to drop if topics were deleted or if you changed the configuration of an existing topic or the full broker (e.g. reduce the log retention time or cause the logs to roll over less frequently), and to go up if you change the configuration in the opposite direction.

这篇关于卡夫卡vm.max_map_count的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！