Maven的spark-submit ClassNotFound异常 [英] spark-submit ClassNotFound Exception with Maven

问题描述

我意识到与此相关的问题，但是我无法使我的代码正常工作.

I realize there are related questions with this one, but I just can't get my code to work.

我正在以独立模式运行Spark Streaming应用程序，主节点位于Windows主机中，而工作节点位于Ubuntu虚拟机中. 问题出在这里:运行spark-submit时，显示的是这样:

I am running a Spark Streaming application in standalone mode, with the master node in my Windows host and a worker in an Ubuntu virtual machine. Here is the problem: when I run spark-submit, this is what shows up:

 >spark-submit --master spark://192.168.56.1:7077 --class spark.example.Main  C:/Users/Manuel Mourato/xxx/target/ParkMonitor-1.0-SNAPSHOT.jar
Warning: Skip remote jar C:/Users/Manuel.
java.lang.ClassNotFoundException: spark.example.Main
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
    at   org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我使用IntelliJ中的"package"和Maven一起创建了这个jar文件. 这就是我设置SparkStreaming上下文的方式:

I created this jar file with Maven, using "package" in IntelliJ. This is how I am setting up my SparkStreaming Context:

 SparkConf sparkConfiguration= new SparkConf().setAppName("ParkingDataAnalysis").setMaster("spark://192.168.56.1:7077");
 JavaStreamingContext sparkStrContext=new JavaStreamingContext(sparkConfiguration, Durations.seconds(1));

有人可以帮助我吗?非常感谢.

Can anyone help me? Thank you so much.

推荐答案

文件夹名称中有空格，请添加引号()，然后重试

You've got space in folder name, please add quotes (") and try again

spark-submit --master spark://192.168.56.1:7077 --class spark.example.Main "C:/Users/Manuel Mourato/xxx/target/ParkMonitor-1.0-SNAPSHOT.jar"

另一个，来自文档:a pplication-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

One more, from docs: application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

因此，请将文件复制到HDFS或所有节点上的相同位置.结合使用Linux和Windows会很困难；)我强烈建议您设置HDFS

So please copy your file to HDFS or to the same location on all nodes. It would be difficult in combination of Linux and Windows ;) I strongly recommend setting up HDFS

这篇关于Maven的spark-submit ClassNotFound异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！