Apache Spark-使用spark-submit抛出NoSuchMethodError [英] Apache Spark -- using spark-submit throws a NoSuchMethodError

问题描述

要将Spark应用程序提交到集群，其文档说明:

To submit a Spark application to a cluster, their documentation notes:

要执行此操作，请创建一个包含您的代码及其依赖项的程序集jar(或超级" jar). sbt和Maven都有程序集插件.创建程序集jar时，将Spark和Hadoop列为提供的依赖项；这些不需要捆绑在一起，因为它们是由集群管理器在运行时提供的. - http://spark.apache.org/docs/latest/submitting-applications .html

因此，我将Apache Maven Shade插件添加到了pom.xml文件中. (版本3.0.0)
然后，我将Spark依赖项的范围转换为provided. (版本2.1.0)

So, I added the Apache Maven Shade Plugin to my pom.xml file. (version 3.0.0)
And I turned my Spark dependency's scope into provided. (version 2.1.0)

(我还添加了Apache Maven Assembly Plugin，以确保我在运行mvn clean package时将所有依赖项包装在jar中.我不确定这是否确实必要.)

(I also added the Apache Maven Assembly Plugin to ensure I was wrapping all of my dependencies in the jar when I run mvn clean package. I'm unsure if it's truly necessary.)

这就是spark-submit失败的原因.它为我具有的依赖项抛出NoSuchMethodError(请注意，假设删除了provided，则在IntelliJ中进行编译时，代码将从本地实例运行).

Thus is how spark-submit fails. It throws a NoSuchMethodError for a dependency I have (note that the code works from a local instance when compiling inside IntelliJ, assuming that provided is removed).

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;

引发错误的代码行是无关紧要的-它仅仅是我的main方法中创建Stopwatch的第一行，这是Google Guava实用程序的一部分. (版本21.0)

The line of code that throws the error is irrelevant--it's simply the first line in my main method that creates a Stopwatch, part of the Google Guava utilities. (version 21.0)

其他在线解决方案表明，它与Guava的版本冲突有关，但是我对这些建议还没有碰运气.任何帮助将不胜感激，谢谢.

Other solutions online suggest that it has to do with version conflicts of Guava, but I haven't had any luck yet with those suggestions. Any help would be appreciated, thank you.

推荐答案

如果查看Spark 2.1.0安装的/jars子目录，则可能会看到guava-14.0.1.jar.根据 API对于您正在使用的Guava Stopwatch#createStarted方法，直到Guava 15.0才存在createStarted.最有可能发生的事情是，Spark进程Classloader在找到打包在您的uberjar中的Guava 21.0库之前，先找到了Spark提供的Guava 14.0.1库.

If you take a look at the /jars subdirectory of the Spark 2.1.0 installation, you will likely see guava-14.0.1.jar. Per the API for the Guava Stopwatch#createStarted method you are using, createStarted did not exist until Guava 15.0. What is most likely happening is that the Spark process Classloader is finding the Spark-provided Guava 14.0.1 library before it finds the Guava 21.0 library packaged in your uberjar.

一种可能的解决方法是使用 class-relocation Maven Shade插件提供的功能(您已经在使用它来构建uberjar).通过类重定位"，Maven-Shade在打包uberjar的过程中将Guava 21.0类(您的代码需要)从pattern位置(反映其现有包名称(例如com.google.common.base))移动到任意shadedPattern位置，您可以在Shade配置中指定该值(例如myguava123.com.google.common.base).

One possible resolution is to use the class-relocation feature provided by the Maven Shade plugin (which you're already using to construct your uberjar). Via "class relocation", Maven-Shade moves the Guava 21.0 classes (needed by your code) during the packaging of the uberjar from a pattern location reflecting their existing package name (e.g. com.google.common.base) to an arbitrary shadedPattern location, which you specify in the Shade configuration (e.g. myguava123.com.google.common.base).

结果是，较新的Guava库不再共享包名，从而避免了运行时冲突.

The result is that the older and newer Guava libraries no longer share a package name, avoiding the runtime conflict.

这篇关于Apache Spark-使用spark-submit抛出NoSuchMethodError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！