Apache Spark-使用spark-submit抛出NoSuchMethodError [英] Apache Spark -- using spark-submit throws a NoSuchMethodError
问题描述
要将Spark应用程序提交到集群,其文档说明:
To submit a Spark application to a cluster, their documentation notes:
要执行此操作,请创建一个包含您的代码及其依赖项的程序集jar(或超级" jar). sbt和Maven都有程序集插件.创建程序集jar时,将Spark和Hadoop列为提供的依赖项;这些不需要捆绑在一起,因为它们是由集群管理器在运行时提供的. - http://spark.apache.org/docs/latest/submitting-applications .html
因此,我将Apache Maven Shade插件添加到了pom.xml
文件中. (版本3.0.0)
然后,我将Spark依赖项的范围转换为provided
. (版本2.1.0)
So, I added the Apache Maven Shade Plugin to my pom.xml
file. (version 3.0.0)
And I turned my Spark dependency's scope into provided
. (version 2.1.0)
(我还添加了Apache Maven Assembly Plugin,以确保我在运行mvn clean package
时将所有依赖项包装在jar中.我不确定这是否确实必要.)
(I also added the Apache Maven Assembly Plugin to ensure I was wrapping all of my dependencies in the jar when I run mvn clean package
. I'm unsure if it's truly necessary.)
这就是spark-submit
失败的原因.它为我具有的依赖项抛出NoSuchMethodError
(请注意,假设删除了provided
,则在IntelliJ中进行编译时,代码将从本地实例运行).
Thus is how spark-submit
fails. It throws a NoSuchMethodError
for a dependency I have (note that the code works from a local instance when compiling inside IntelliJ, assuming that provided
is removed).
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;
引发错误的代码行是无关紧要的-它仅仅是我的main方法中创建Stopwatch
的第一行,这是Google Guava实用程序的一部分. (版本21.0)
The line of code that throws the error is irrelevant--it's simply the first line in my main method that creates a Stopwatch
, part of the Google Guava utilities. (version 21.0)
其他在线解决方案表明,它与Guava的版本冲突有关,但是我对这些建议还没有碰运气.任何帮助将不胜感激,谢谢.
Other solutions online suggest that it has to do with version conflicts of Guava, but I haven't had any luck yet with those suggestions. Any help would be appreciated, thank you.
推荐答案
如果查看Spark 2.1.0安装的/jars
子目录,则可能会看到guava-14.0.1.jar
.根据 API对于您正在使用的Guava Stopwatch#createStarted
方法,直到Guava 15.0才存在createStarted
.最有可能发生的事情是,Spark进程Classloader在找到打包在您的uberjar中的Guava 21.0库之前,先找到了Spark提供的Guava 14.0.1库.
If you take a look at the /jars
subdirectory of the Spark 2.1.0 installation, you will likely see guava-14.0.1.jar
. Per the API for the Guava Stopwatch#createStarted
method you are using, createStarted
did not exist until Guava 15.0. What is most likely happening is that the Spark process Classloader is finding the Spark-provided Guava 14.0.1 library before it finds the Guava 21.0 library packaged in your uberjar.
一种可能的解决方法是使用 class-relocation Maven Shade插件提供的功能(您已经在使用它来构建uberjar).通过类重定位",Maven-Shade在打包uberjar的过程中将Guava 21.0类(您的代码需要)从pattern
位置(反映其现有包名称(例如com.google.common.base
))移动到任意shadedPattern
位置,您可以在Shade配置中指定该值(例如myguava123.com.google.common.base
).
One possible resolution is to use the class-relocation feature provided by the Maven Shade plugin (which you're already using to construct your uberjar). Via "class relocation", Maven-Shade moves the Guava 21.0 classes (needed by your code) during the packaging of the uberjar from a pattern
location reflecting their existing package name (e.g. com.google.common.base
) to an arbitrary shadedPattern
location, which you specify in the Shade configuration (e.g. myguava123.com.google.common.base
).
结果是,较新的Guava库不再共享包名,从而避免了运行时冲突.
The result is that the older and newer Guava libraries no longer share a package name, avoiding the runtime conflict.
这篇关于Apache Spark-使用spark-submit抛出NoSuchMethodError的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!