星火广播错误：超过spark.akka.frameSize考虑使用广播 [英] Spark broadcast error: exceeds spark.akka.frameSize Consider using broadcast

问题描述

我有一个大的数据称为边缘

I have a large data called "edges"

org.apache.spark.rdd.RDD[org.apache.spark.graphx.Edge[(String, Int)]] = MappedRDD[27] at map at <console>:52

当我在单机模式下是工作，我是能够收集，统计和保存此文件。现在，在群集上，我得到这个错误

When I was working in standalone mode, I was able to collect, count and save this file. Now, on a cluster, I'm getting this error

edges.count
...
Serialized task 28:0 was 12519797 bytes which exceeds spark.akka.frameSize
  (10485760 bytes). Consider using broadcast variables for large values.

同样的，.saveAsTextFile（边缘）

Same with .saveAsTextFile("edges")

这是从火花塞壳。我已经使用选项尝试结果
    --driver-java的选项-Dspark.akka.frameSize = 15

This is from the spark-shell. I have tried using the option
--driver-java-options "-Dspark.akka.frameSize=15"

但是，当我做到这一点，它只是挂起下去。任何帮助将是AP preciated。

But when I do that, it just hangs indefinitely. Any help would be appreciated.

**编辑**

我的独立模式是在星火1.1.0和我的集群星火1.0.1。

My standalone mode was on Spark 1.1.0 and my cluster is Spark 1.0.1.

此外，悬发生时，我去算，收集或*的saveAs的RDD，但将其定义或做滤镜在它工作得很好。

Also, the hanging occurs when I go to count, collect or saveAs* the RDD, but defining it or doing filters on it work just fine.

推荐答案

在考虑使用广播变量值较大的错误消息通常表明您已经捕捉到函数闭包一些大的变数。例如，您可能已经编写类似于

The "Consider using broadcast variables for large values" error message usually indicates that you've captured some large variables in function closures. For example, you might have written something like

val someBigObject = ...
rdd.mapPartitions { x => doSomething(someBigObject, x) }.count()

这将导致 someBigObject 来捕获和系列化与你的任务。如果你正在做这样的事情，你可以使用广播变量代替，这将导致只有一个参考对象将被存储在任务本身，而实际的对象数据将被分别发送。

which causes someBigObject to be captured and serialized with your task. If you're doing something like that, you can use a broadcast variable instead, which will cause only a reference to the object to be stored in the task itself, while the actual object data will be sent separately.

在火花1.1.0+，它不是严格必要使用用于该广播变量，因为任务将自动播放（参见有关详细信息，SPARK-2521 ）。目前仍有理由使用广播变量（如共享跨多个动作/作业的大对象），但你不会的需求的使用，以避免帧大小的错误。

In Spark 1.1.0+, it isn't strictly necessary to use broadcast variables for this, since tasks will automatically be broadcast (see SPARK-2521 for more details). There are still reasons to use broadcast variables (such as sharing a big object across multiple actions / jobs), but you won't need to use it to avoid frame size errors.

另一种选择是增加阿卡帧的大小。在任何星火版本，你应该能够设置 spark.akka.frameSize 在 SparkConf 设置，然后再创建SparkContext。正如你可能已经注意到了，不过，这是在火花壳，其中为您创建上下文有点困难。在火花（1.1.0或更高版本）的新版本，你可以通过 - 配置spark.akka.frameSize = 16 启动时火花壳。在星火1.0.1或1.0.2，你应该能够通过 - 驱动程序的Java选项-Dspark.akka.frameSize = 16而不是

Another option is to increase the Akka frame size. In any Spark version, you should be able to set the spark.akka.frameSize setting in SparkConf prior to creating your SparkContext. As you may have noticed, though, this is a little harder in spark-shell, where the context is created for you. In newer versions of Spark (1.1.0 and higher), you can pass --conf spark.akka.frameSize=16 when launching spark-shell. In Spark 1.0.1 or 1.0.2, you should be able to pass --driver-java-options "-Dspark.akka.frameSize=16" instead.

这篇关于星火广播错误：超过spark.akka.frameSize考虑使用广播的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！