为什么带有扇贝的Scala 2.11和Spark会导致"java.lang.NoSuchMethodError:scala.reflect.api.JavaUniverse.runtimeMirror"? [英] Why do Scala 2.11 and Spark with scallop lead to "java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror"?

问题描述

我正在使用Scala 2.11，Spark和扇贝( https://github.com/scallop/scallop ).我使用sbt构建了没有Spark提供的依赖项的应用胖子(位于:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar)

I am using Scala 2.11, Spark, and Scallop (https://github.com/scallop/scallop). I used sbt to build an application fat jar without Spark provided dependencies (this is at: analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar)

我可以在sbt中正常运行该程序.

I am able to run the program fine in sbt.

我试图从命令行运行它，如下所示:

I tried to run it from the command line as follows:

time ADD_JARS=analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar java -cp /Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar com.dtex.analysis.transform.GenUserSummaryView -d /Users/arun/DataSets/LME -p output -s txt -o /Users/arun/tmp/LME/LME

我收到以下错误消息:

线程主"中的异常java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse $ JavaMirror; 在org.rogach.scallop.package $.(package.scala:37)在 org.rogach.scallop.package $.(package.scala)在 com.dtex.analysis.transform.GenUserSummaryView $ Conf.delayedEndpoint $ com $ dtex $ analysis $ transform $ GenUserSummaryView $ Conf $ 1(GenUserSummaryView.scala:27) 在 com.dtex.analysis.transform.GenUserSummaryView $ Conf $ delayedInit $ body.apply(GenUserSummaryView.scala:26) 在scala.Function0 $ class.apply $ mcV $ sp(Function0.scala:40)在 scala.runtime.AbstractFunction0.apply $ mcV $ sp(AbstractFunction0.scala:12) 在org.rogach.scallop.AfterInit $ class.delayedInit(AfterInit.scala:12) 在org.rogach.scallop.ScallopConf.delayedInit(ScallopConf.scala:26) 在 com.dtex.analysis.transform.GenUserSummaryView $ Conf.(GenUserSummaryView.scala:26) 在 com.dtex.analysis.transform.GenUserSummaryView $ .main(GenUserSummaryView.scala:54) 在 com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)

Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror; at org.rogach.scallop.package$.(package.scala:37) at org.rogach.scallop.package$.(package.scala) at com.dtex.analysis.transform.GenUserSummaryView$Conf.delayedEndpoint$com$dtex$analysis$transform$GenUserSummaryView$Conf$1(GenUserSummaryView.scala:27) at com.dtex.analysis.transform.GenUserSummaryView$Conf$delayedInit$body.apply(GenUserSummaryView.scala:26) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at org.rogach.scallop.AfterInit$class.delayedInit(AfterInit.scala:12) at org.rogach.scallop.ScallopConf.delayedInit(ScallopConf.scala:26) at com.dtex.analysis.transform.GenUserSummaryView$Conf.(GenUserSummaryView.scala:26) at com.dtex.analysis.transform.GenUserSummaryView$.main(GenUserSummaryView.scala:54) at com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)

推荐答案

问题是您使用了不兼容的Scala版本，即Spark是用Scala 2.10编译的，而您试图使用Scala 2.11.

The issue is that you've used incompatible Scala versions, i.e. Spark was compiled with Scala 2.10 and you were trying to use Scala 2.11.

将所有内容移至Scala 2.10版本，并确保您也更新了SBT.

Move everything to Scala 2.10 version and make sure you update your SBT as well.

您还可以尝试为Scala 2.11.7编译Spark源，并改用它.

You may also try to compile Spark sources for Scala 2.11.7 and use it instead.

这篇关于为什么带有扇贝的Scala 2.11和Spark会导致"java.lang.NoSuchMethodError:scala.reflect.api.JavaUniverse.runtimeMirror"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！