提交火花时出现 ClassNotFoundException scala.runtime.LambdaDeserialize [英] ClassNotFoundException scala.runtime.LambdaDeserialize when spark-submit
本文介绍了提交火花时出现 ClassNotFoundException scala.runtime.LambdaDeserialize的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
I follow the Scala tutorial on https://spark.apache.org/docs/2.1.0/quick-start.html
My scala file
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp {
def main(args: Array[String]) {
val logFile = "/data/README.md" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println(s"Lines with a: $numAs, Lines with b: $numBs")
sc.stop()
}
}
and build.sbt
name := "Simple Project"
version := "1.0"
scalaVersion := "2.12.4"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0"
I ran sbt package
sucessfully (already delete everything except scala source code and build.sbt then run sbt package
again)
[info] Loading project definition from /home/cpu11453local/workspace/testspark_scala/project
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/my_name/workspace/testspark_scala/)
[info] Packaging /home/my_name/workspace/testspark_scala/target/scala-2.12/simple-project_2.12-1.0.jar ...
[info] Done packaging.
[success] Total time: 1 s, completed Nov 8, 2017 12:15:24 PM
However, when I run spark submit
$SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] simple-project_2.12-1.0.jar
I got error
java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
Full spark-submit output on gist
解决方案
as @Alexey said, change Scala version to 2.11 fixed the problem.
build.sbt
name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.11"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"
Note that Scala version MUST MATCH with Spark. Look at the artifactId, spark-core_2.11 mean it was compatible with scala 2.11 (No backward or forward compatible)
这篇关于提交火花时出现 ClassNotFoundException scala.runtime.LambdaDeserialize的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文