值"spark.yarn.executor.memoryOverhead"设置为"spark.yarn.executor.memoryOverhead".环境? [英] The value of "spark.yarn.executor.memoryOverhead" setting?

问题描述

在具有YARN的Spark作业中，spark.yarn.executor.memoryOverhead的值应该分配给App还是仅分配给最大值?

The value of spark.yarn.executor.memoryOverhead in a Spark job with YARN should be allocated to App or just the max value?

推荐答案

spark.yarn.executor.memoryOverhead

仅仅是最大值.目标是将RDHEAD和数据帧所使用的OVERHEAD计算为实际执行程序内存的百分比

Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames

--executor-memory/spark.executor.memory

控制执行程序堆的大小，但是JVM也可以在堆外使用一些内存，例如，用于内联的String和直接字节缓冲区.

controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers.

spark.yarn.executor.memoryOverhead的值 属性被添加到执行者内存中，以确定每个执行者对YARN的完整内存请求.默认值为max(executorMemory * 0.10，最小为384).

The value of the spark.yarn.executor.memoryOverhead property is added to the executor memory to determine the full memory request to YARN for each executor. It defaults to max(executorMemory * 0.10, with minimum of 384).

执行者将使用基于spark.executor.memory属性的内存分配以及由spark.yarn.executor.memoryOverhead

The executors will use a memory allocation based on the property of spark.executor.memoryplus an overhead defined by spark.yarn.executor.memoryOverhead

这篇关于值"spark.yarn.executor.memoryOverhead"设置为"spark.yarn.executor.memoryOverhead".环境?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！