Spark 故障:引起:org.apache.spark.shuffle.FetchFailedException:框架太大:5454002341 [英] Spark Failure : Caused by: org.apache.spark.shuffle.FetchFailedException: Too large frame: 5454002341
问题描述
我正在为确定父子项的表生成层次结构.
I am generating a hierarchy for a table determining the parent child.
以下是使用的配置,即使在收到关于过大框架的错误后:
Below is the configuration used, even after getting the error with regards to the too large frame:
--conf spark.yarn.executor.memoryOverhead=1024mb \
--conf yarn.nodemanager.resource.memory-mb=12288mb \
--driver-memory 32g \
--driver-cores 8 \
--executor-cores 32 \
--num-executors 8 \
--executor-memory 256g \
--conf spark.maxRemoteBlockSizeFetchToMem=15g
<小时>
import org.apache.log4j.{Level, Logger};
import org.apache.spark.SparkContext;
import org.apache.spark.sql.{DataFrame, SparkSession};
import org.apache.spark.sql.functions._;
import org.apache.spark.sql.expressions._;
lazy val sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate();
import spark.implicits._;
val hiveEmp: DataFrame = sparkSession.sql("select * from db.employee");
hiveEmp.repartition(300);
import org.apache.spark.sql.functions._;
val nestedLevel = 3;
val empHierarchy = (1 to nestedLevel).foldLeft(hiveEmp.as("wd0")) { (wDf, i) =>
val j = i - 1
wDf.join(hiveEmp.as(s"wd$i"), col(s"wd$j.parent_id".trim) === col(s"wd$i.id".trim), "left_outer")
}.select(
col("wd0.id") :: col("wd0.parent_id") ::
col("wd0.amount").as("amount") :: col("wd0.payment_id").as("payment_id") :: (
(1 to nestedLevel).toList.map(i => col(s"wd$i.amount").as(s"amount_$i")) :::
(1 to nestedLevel).toList.map(i => col(s"wd$i.payment_id").as(s"payment_id_$i"))
): _*);
empHierarchy.write.saveAsTable("employee4");
错误
Caused by: org.apache.spark.SparkException: Task failed while writing rows
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
... 3 more
Caused by: org.apache.spark.shuffle.FetchFailedException: Too large frame: 5454002341
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:361)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:336)
推荐答案
Suresh 是对的.这是一个更好的记录 &他的答案的格式化版本以及一些有用的背景信息:
Suresh is right. Here's a better documented & formatted version of his answer with some useful background info:
- bug report (link to the fix is at the very bottom)
- fix (fixed as of 2.2.0 - already mentioned by Jared)
- change of config's default value (changed as of 2.4.0)
如果您使用的是 2.2.x 或 2.3.x 版本,则可以通过将 config 的值设置为 Int.MaxValue - 512
来达到相同的效果,即通过设置 spark.maxRemoteBlockSizeFetchToMem=2147483135
.请参阅此处了解 默认值,截至 2019 年 9 月使用.
If you're on a version 2.2.x or 2.3.x, you can achieve the same effect by setting the value of the config to Int.MaxValue - 512
, i.e. by setting spark.maxRemoteBlockSizeFetchToMem=2147483135
. See here for the default value used as of September 2019.
这篇关于Spark 故障:引起:org.apache.spark.shuffle.FetchFailedException:框架太大:5454002341的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!