星火核1.6.1& lift-json 2.6.3 java.lang.NoClassDefFoundError [英] spark-core 1.6.1 & lift-json 2.6.3 java.lang.NoClassDefFoundError

问题描述

我有一个Spark应用程序，其中包含一个sbt文件，如下所示.
它可以在我的本地计算机上运行.但是，当我将其提交给运行Spark 1.6.1的EMR时，发生了如下错误:

I have a Spark application which has a sbt file just like below.
It works on my local machine. But when I submit it to EMR running Spark 1.6.1, an error occured like below:

java.lang.NoClassDefFoundError: net/liftweb/json/JsonAST$JValue

我正在使用"sbt-package"来获取jar

Build.sbt:

I am using "sbt-package" to get jar

Build.sbt:

organization := "com.foo"
name := "FooReport"

version := "1.0"

scalaVersion := "2.10.6"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.6.1"
  ,"net.liftweb" % "lift-json_2.10" % "2.6.3"
  ,"joda-time" % "joda-time" % "2.9.4"
)

您对发生的事情有任何了解吗?

Do you have any idea about what’s happening?

推荐答案

我已经找到了一种解决方案，并且可以正常工作！

I' ve found a solution and it is working!

问题全部出在sbt package上，它不包括所有依赖jar到输出jar.为了克服这个问题，我尝试了sbt-assembly，但是运行它时出现了很多" deduplicate "错误.

The problem was all about sbt package which doesn't include all dependent jars to output jar. To overcome this I tried sbt-assembly but I got plenty of "deduplicate" errors when I ran it.

毕竟，我来到了这篇博客文章，这一切都变得很清楚.
http://queirozf .com/entries/creating-scala-fat-jars-spark-on-sbt-with-sbt-assembly-plugin

After all I came up to this blog post which made everthing clear.
http://queirozf.com/entries/creating-scala-fat-jars-for-spark-on-sbt-with-sbt-assembly-plugin

为了将Spark作业提交到Spark集群(通过spark-submit)， 您需要在其中包括所有依赖项(Spark本身除外) Jar，否则您将无法在工作中使用它们.

In order to submit Spark jobs to a Spark Cluster (via spark-submit), you need to include all dependencies (other than Spark itself) in the Jar, otherwise you won't be able to use those in your job.

1. 在/project文件夹下创建"assembly.sbt".
   
2. 添加此行addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
3. 然后将下面的assemblyMergeStrategy代码粘贴到您的 build.sbt

1. Create "assembly.sbt" under /project folder.
   
2. Add this line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
3. Then paste the assemblyMergeStrategy code below to your build.sbt

assemblyMergeStrategy in assembly := { case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last case PathList("javax", "activation", xs @ _*) => MergeStrategy.last case PathList("org", "apache", xs @ _*) => MergeStrategy.last case PathList("com", "google", xs @ _*) => MergeStrategy.last case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last case PathList("com", "codahale", xs @ _*) => MergeStrategy.last case PathList("com", "yammer", xs @ _*) => MergeStrategy.last case "about.html" => MergeStrategy.rename case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last case "META-INF/mailcap" => MergeStrategy.last case "META-INF/mimetypes.default" => MergeStrategy.last case "plugin.properties" => MergeStrategy.last case "log4j.properties" => MergeStrategy.last case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }

assemblyMergeStrategy in assembly := { case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last case PathList("javax", "activation", xs @ _*) => MergeStrategy.last case PathList("org", "apache", xs @ _*) => MergeStrategy.last case PathList("com", "google", xs @ _*) => MergeStrategy.last case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last case PathList("com", "codahale", xs @ _*) => MergeStrategy.last case PathList("com", "yammer", xs @ _*) => MergeStrategy.last case "about.html" => MergeStrategy.rename case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last case "META-INF/mailcap" => MergeStrategy.last case "META-INF/mimetypes.default" => MergeStrategy.last case "plugin.properties" => MergeStrategy.last case "log4j.properties" => MergeStrategy.last case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }

并运行sbt assembly

现在您有一个带有所有依赖项的大胖子.基于从属库，可能为数百MB.就我而言，我使用的是已经安装了Spark 1.6.1的AWS EMR.要从jar中排除spark-core lib，可以使用提供的"关键字:

Now you have a big fat jar which has all the dependencies. It might be hundreds of MB based on dependent libraries. For my case I am using Aws EMR which Spark 1.6.1 is already installed on it. To exclude spark-core lib from your jar you can use "provided" keyword:

"org.apache.spark" %% "spark-core" % "1.6.1" % "provided"

这是最终的build.sbt文件:

Here is the final build.sbt file:

organization := "com.foo"
name := "FooReport"

version := "1.0"

scalaVersion := "2.10.6"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.6.1" % "provided"
  ,"net.liftweb" % "lift-json_2.10" % "2.6.3"
  ,"joda-time" % "joda-time" % "2.9.4"
)

assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.last
  case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
  case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
  case "META-INF/mailcap" => MergeStrategy.last
  case "META-INF/mimetypes.default" => MergeStrategy.last
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

这篇关于星火核1.6.1& lift-json 2.6.3 java.lang.NoClassDefFoundError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！