了解spark.yarn.executor.memoryOverhead [英] Understanding spark.yarn.executor.memoryOverhead
问题描述
当我在yarn上运行spark应用程序时，驱动程序和执行程序的内存设置为--driver-memory 4G --executor-memory 2G
When I am running a spark application on yarn, with driver and executor memory settings as --driver-memory 4G --executor-memory 2G
然后，当我运行应用程序时，抛出异常，抱怨说Container killed by YARN for exceeding memory limits. 2.5 GB of 2.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
Then when I run the application, an exceptions throws complaining that Container killed by YARN for exceeding memory limits. 2.5 GB of 2.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
这2.5 GB在这里是什么意思? (开销内存，执行程序内存还是开销+执行程序内存?)之所以这样问，是因为当我将内存设置更改为:
What does this 2.5 GB mean here? (overhead memory, executor memory or overhead+executor memory?)I ask so because when I change the the memory settings as:
--driver-memory 4G --executor-memory 4G --conf --driver-memory 4G --conf spark.yarn.executor.memoryOverhead=2048，那么异常消失了.
我会问，尽管我将开销内存提高到了2G，但仍低于2.5G，为什么现在可以正常工作?
I would ask, although I have boosted the overhead memory to 2G, it is still under 2.5G, why does it work now?
推荐答案
让我们了解如何在spark中的各个区域之间分配内存.
Let us understand how memory is divided among various regions in spark.
- 执行器MemoryOverhead:
spark.yarn.executor.memoryOverhead = max(384 MB, .07 * spark.executor.memory).
在第一种情况下，memoryOverhead = max(384 MB, 0.07 * 2 GB) = max(384 MB, 143.36 MB)因此，假设您为每个执行者分配了单个内核，则在每个执行者中都保留memoryOverhead = 384 MB.
spark.yarn.executor.memoryOverhead = max(384 MB, .07 * spark.executor.memory).
In your first case, memoryOverhead = max(384 MB, 0.07 * 2 GB) = max(384 MB, 143.36 MB) Hence, memoryOverhead = 384 MB is reserved in each executer assuming you have assigned single core per executer.
- 执行和存储内存:
默认情况下为spark.memory.fraction = 0.6，这意味着执行和存储作为统一区域占据了剩余内存的60％，即998 MB.除非启用spark.memory.useLegacyMode，否则没有严格的边界分配给每个区域.否则，它们将共享移动边界.
By default spark.memory.fraction = 0.6, which implies that execution and storage as a unified region occupy 60% of the remaining memory i.e. 998 MB. There is no strict boundary that is allocated to each region unless you enable spark.memory.useLegacyMode. Otherwise they share a moving boundary.
- 用户内存:
分配执行和存储内存后仍保留的内存池，完全取决于您以自己喜欢的方式使用它.您可以在其中存储将在RDD转换中使用的数据结构.例如，您可以通过使用mapPartitions转换来维护此哈希表的哈希表来重写Spark聚合.这包括MemoryOverhead之后剩余的40％的剩余内存.您的情况是~660 MB.
Memory pool that remains after the allocation of Execution and Storage Memory, and it is completely up to you to use it in a way you like. You can store your own data structures there that would be used in RDD transformations. For example, you can rewrite Spark aggregation by using mapPartitions transformation maintaining hash table for this aggregation to run. This comprises the rest of 40% memory left after MemoryOverhead. In your case it is ~660 MB.
如果您的工作没有满足上述任何分配条件，那么很有可能最终导致OOM问题.
If any of the above allocations are not met by your job, then it is highly likely to end up in OOM problems.
这篇关于了解spark.yarn.executor.memoryOverhead的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
