为什么 Scala 2.11 和带有扇贝的 Spark 会导致“java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror"? [英] Why do Scala 2.11 and Spark with scallop lead to "java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror"?

问题描述

我正在使用 Scala 2.11、Spark 和 Scallop(https://github.com/scallop/scallop).我使用 sbt 构建了一个没有 Spark 提供的依赖项的应用程序 fat jar(位于:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar)

I am using Scala 2.11, Spark, and Scallop (https://github.com/scallop/scallop). I used sbt to build an application fat jar without Spark provided dependencies (this is at: analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar)

我可以在 sbt 中正常运行程序.

I am able to run the program fine in sbt.

我尝试从命令行运行它，如下所示:

I tried to run it from the command line as follows:

time ADD_JARS=analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar java -cp /Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar com.dtex.analysis.transform.GenUserSummaryView -d /Users/arun/DataSets/LME -p output -s txt -o /Users/arun/tmp/LME/LME

我收到以下错误消息:

线程主"java.lang.NoSuchMethodError 中的异常:scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;在 org.rogach.scallop.package$.(package.scala:37) 在org.rogach.scallop.package$.(package.scala) 在com.dtex.analysis.transform.GenUserSummaryView$Conf.delayedEndpoint$com$dtex$analysis$transform$GenUserSummaryView$Conf$1(GenUserSummaryView.scala:27​​)在com.dtex.analysis.transform.GenUserSummaryView$Conf$delayedInit$body.apply(GenUserSummaryView.scala:26)在 scala.Function0$class.apply$mcV$sp(Function0.scala:40) 在scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)在 org.rogach.scallop.AfterInit$class.delayedInit(AfterInit.scala:12)在 org.rogach.scallop.ScallopConf.delayedInit(ScallopConf.scala:26)在com.dtex.analysis.transform.GenUserSummaryView$Conf.(GenUserSummaryView.scala:26)在com.dtex.analysis.transform.GenUserSummaryView$.main(GenUserSummaryView.scala:54)在com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)

Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror; at org.rogach.scallop.package$.(package.scala:37) at org.rogach.scallop.package$.(package.scala) at com.dtex.analysis.transform.GenUserSummaryView$Conf.delayedEndpoint$com$dtex$analysis$transform$GenUserSummaryView$Conf$1(GenUserSummaryView.scala:27) at com.dtex.analysis.transform.GenUserSummaryView$Conf$delayedInit$body.apply(GenUserSummaryView.scala:26) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at org.rogach.scallop.AfterInit$class.delayedInit(AfterInit.scala:12) at org.rogach.scallop.ScallopConf.delayedInit(ScallopConf.scala:26) at com.dtex.analysis.transform.GenUserSummaryView$Conf.(GenUserSummaryView.scala:26) at com.dtex.analysis.transform.GenUserSummaryView$.main(GenUserSummaryView.scala:54) at com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)

推荐答案

问题是你使用了不兼容的 Scala 版本，即 Spark 是用 Scala 2.10 编译的，而你试图使用 Scala 2.11.

The issue is that you've used incompatible Scala versions, i.e. Spark was compiled with Scala 2.10 and you were trying to use Scala 2.11.

将所有内容移至 Scala 2.10 版本，并确保您也更新了 SBT.

Move everything to Scala 2.10 version and make sure you update your SBT as well.

您也可以尝试为 Scala 2.11.7 编译 Spark 源代码并改用它.

You may also try to compile Spark sources for Scala 2.11.7 and use it instead.

这篇关于为什么 Scala 2.11 和带有扇贝的 Spark 会导致“java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！