Spark Shuffle内存错误:无法分配直接内存 [英] spark shuffle memory error: failed to allocate direct memory

问题描述

在spark数据帧(4x)上执行几次联接时，出现以下错误:

When performing a couple of joins on spark data frames (4x) I get the following error:

org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 4294967296, max: 4294967296)

即使在设置时:

--conf "spark.executor.extraJavaOptions-XX:MaxDirectMemorySize=4G" \

它没有解决.

推荐答案

似乎飞行块过多.尝试使用较小的 spark.reducer.maxBlocksInFlightPerAddress 值.作为参考，请查看此 JIRA

Seems like there are too many in flight blocks. Try with smaller values of spark.reducer.maxBlocksInFlightPerAddress. For reference take a look at this JIRA

引用文字:

对于启用了外部随机播放的配置，我们观察到，如果非常大，则不会.从远程主机上获取大量数据块，这会使NM承受更大的压力，并可能使其崩溃.此更改引入了配置spark.reducer.maxBlocksInFlightPerAddress，以限制编号.从给定的远程地址获取的地图输出的数量.此处应用的更改适用于两种情况-启用和禁用外部随机播放.

For configurations with external shuffle enabled, we have observed that if a very large no. of blocks are being fetched from a remote host, it puts the NM under extra pressure and can crash it. This change introduces a configuration spark.reducer.maxBlocksInFlightPerAddress , to limit the no. of map outputs being fetched from a given remote address. The changes applied here are applicable for both the scenarios - when external shuffle is enabled as well as disabled.

这篇关于Spark Shuffle内存错误:无法分配直接内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！