使用 Spark Submit 执行 Spring Boot 应用程序时出现 ClassNotFoundException [英] ClassNotFoundException when executing Spring Boot Application Using Spark Submit
本文介绍了使用 Spark Submit 执行 Spring Boot 应用程序时出现 ClassNotFoundException的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试使用 spark-submit 执行我的 Spring Boot 示例应用程序,但它抛出了 ClassNotFoundException
.
I am trying to execute my Spring Boot Sample Application using spark-submit but it is throwing ClassNotFoundException
.
java.lang.NoClassDefFoundError: org/springframework/boot/SpringApplication
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.SpringApplication
我也在我的 pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.4.RELEASE</version>
</parent>
<dependencies>
<dependency> <!-- Spark -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
</dependencies>
我的示例代码是:
@SpringBootApplication
public class dummy {
public static void main (String[] args){
SpringApplication.run(dummy.class, args);
SparkConf conf = new SparkConf().setAppName("sparkSample").set("spark.executor.heartbeatInterval", "700000")
.set("spark.network.timeout", "1000000").set("spark.files.fetchTimeout", "800000");
JavaSparkContext sc = new JavaSparkContext(conf);
System.out.println("normal code...........................");
sc.stop();
}`
我的火花提交命令是:
./spark-submit \
--class sample.dummy \
--master yarn-cluster \
--driver-memory 5G \
--deploy-mode cluster \
--executor-memory 7G \
--executor-cores 1 \
--num-executors 1 \
--conf spark.yarn.executor.memoryOverhead=1024 \
/hadoop/app/abc/bootup/sample-0.0.1-SNAPSHOT.jar \
--runner=SparkRunner
可能是什么问题?
推荐答案
我使用SHADE maven插件解决了.
I solved it by using SHADE maven plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>1.5.2.RELEASE</version>
</dependency>
</dependencies>
<configuration>
<keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.yourpackage.MainClass</mainClass>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
这篇关于使用 Spark Submit 执行 Spring Boot 应用程序时出现 ClassNotFoundException的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文