火花执行器的内存减少到1/2 [英] spark executor memory cut to 1/2

问题描述

我正在做这样的火花提交 spark-submit --class com.mine.myclass --master yarn-cluster --num-executors 3 --executor-memory 4G spark-examples_2.10-1.0.jar

在Web ui中，我可以看到确实有3个执行程序节点，但是每个节点都有2G的内存.当我设置--executor-memory 2G时，ui会显示每个节点1G.

如何将我的设置减少1/2?

解决方案

Web UI的执行程序页面显示了存储内存量，默认情况下，该内存等于Java堆的54％(spark.storage.safetyFraction 0.9 * spark.storage.memoryFraction 0.6)

I am doing a spark-submit like this spark-submit --class com.mine.myclass --master yarn-cluster --num-executors 3 --executor-memory 4G spark-examples_2.10-1.0.jar

in the web ui, I can see indeed there are 3 executor nodes, but each has 2G of memory. When I set --executor-memory 2G, then ui shows 1G per node.

How did it figure to reduce my setting by 1/2?

解决方案

The executor page of the Web UI is showing the amount of storage memory, which is equal to 54% of Java heap by default (spark.storage.safetyFraction 0.9 * spark.storage.memoryFraction 0.6)

这篇关于火花执行器的内存减少到1/2的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！