试图运行样品应用火花提交,但得到找不到类错误时抛出 [英] trying to run sample app with spark-submit but getting class not found excpetion
问题描述
我想从<一个运行的例子href=\"https://github.com/databricks/learning-spark/blob/master/src/main/scala/com/oreilly/learningsparkexamples/scala/StreamingLogInput.scala\"相对=nofollow>学习火花书
包com。示例进口org.apache.spark.SparkConf
进口org.apache.spark.streaming.dstream.DStream
进口org.apache.spark.streaming {秒,的StreamingContext}
进口org.apache.spark._
进口org.apache.spark.SparkContext._
进口org.apache.spark.streaming._
进口org.apache.spark.streaming.dstream._反对SimpleExample {
高清主(参数:数组[字符串]){
VAL主= ARGS(0)
VAL的conf =新SparkConf()。setMaster(主).setAppName(StreamingLogInput)
//创建以1第二批大小的StreamingContext
VAL SSC =新的StreamingContext(CONF,秒(1))
//在端口7777所有输入创建DSTREAM
VAL线= ssc.socketTextStream(本地主机,7777)
VAL errorLines = processLines(系)
//打印出有错误的线路,这将导致此DSTREAM进行评估
errorLines.print()
//开始我们的流上下文,并等待其完成
ssc.start()
//等待10秒,然后退出。要永远跑不叫超时
ssc.awaitTerminationOrTimeout(10000)
ssc.stop()
}
高清processLines(行:DSTREAM [字符串])= {
//我们的过滤器为DSTREAM与错误的行
lines.filter(_。包含(错误))
}}
使用maven项目这是我的pom.xml
&LT;项目&GT;
&LT;&的groupId GT; com.streaming.example&LT; /的groupId&GT;
&LT;&的artifactId GT;流,例如&LT; / artifactId的&GT;
&LT; modelVersion&GT; 4.0.0&LT; / modelVersion&GT;
&LT;名称&gt;&比如LT; /名称&gt;
&LT;包装和GT;&罐子LT; /包装&GT;
&LT;&版GT; 0.0.1&LT; /版本&GT;
&LT;依赖和GT;
&LT;&依赖性GT; &LT;! - 星火依赖 - &GT;
&LT;&的groupId GT; org.apache.spark&LT; /的groupId&GT;
&LT;&的artifactId GT;火花core_2.10&LT; / artifactId的&GT;
&LT;&版GT; 1.1.0&LT; /版本&GT;
&LT;&范围GT;及提供LT; /&范围GT;
&LT; /依赖性&GT;
&LT;&依赖性GT; &LT;! - 星火依赖 - &GT;
&LT;&的groupId GT; org.apache.spark&LT; /的groupId&GT;
&LT;&的artifactId GT;火花streaming_2.10&LT; / artifactId的&GT;
&LT;&版GT; 1.2.0&LT; /版本&GT;
&LT; /依赖性&GT;
&LT; /依赖和GT;
&LT;性状&gt;
&LT; java.version&GT; 1.7 LT; /java.version>
&LT; /性状&gt;
&LT;建立&GT;
&LT; pluginManagement&GT;
&LT;&插件GT;
&LT;&插件GT;
&LT;&的groupId GT; org.apache.maven.plugins&LT; /的groupId&GT;
&LT;&的artifactId GT; Maven的编译器插件&LT; / artifactId的&GT;
&LT;结构&gt;
&lt;信源&GT; 1.6 LT; /源&GT;
&lt;目标&GT; 1.6 LT; /目标与GT;
&LT; /结构&gt;
&LT; /插件&GT;
&LT;&插件GT;
&LT;&的groupId GT; net.alchim31.maven&LT; /的groupId&GT;
&LT;&的artifactId GT;斯卡拉,Maven的插件&LT; / artifactId的&GT;
&LT;&版GT; 3.1.6&LT; /版本&GT;
&LT;&执行GT;
&LT;执行与GT;
&LT;目标&GT;
&LT;&目标GT;&编译LT; /目标&GT;
&LT;&目标GT; testCompile&LT; /目标&GT;
&LT; /目标&GT;
&LT; /执行&GT;
&LT; /处决&GT;
&LT;结构&gt;
&LT;&ARGS GT;
&LT;! - 解决类似https://issues.scala-lang.org/browse/SI-8358 - &GT;
&LT;&ARG GT; -nobootcp&LT; / ARG&GT;
&LT; / ARGS&GT;
&LT; /结构&gt;
&LT; /插件&GT;
&LT; /插件&GT;
&LT; / pluginManagement&GT;
&LT; /构建&GT;
&LT; /项目&GT;
但像这样运行时:
&GT;火花提交--class的src /斯卡拉/ COM /例子/ SimpleExample.scala \\
&GT;目标/流,例如-0.0.1.jar本地[4]
我得到这个错误:
抛出java.lang.ClassNotFoundException:SRC /斯卡拉/ COM /例子/ SimpleExample
在java.lang.Class.forName0(本机方法)
块引用>修改JAR的内容:
$罐子TF目标/流,例如-0.0.1.jar
META-INF /
META-INF / MANIFEST.MF
META-INF /行家/
META-INF /行家/ com.streaming.example /
META-INF /行家/ com.streaming.example /流,例如/
META-INF /行家/ com.streaming.example /流,例如/ pom.xml的
解决方案您SimpleExample.class是不是在你的罐子。
检查Maven构建插件。
您可以考虑使用该组件插件,以及编译:
MVN装配:装配
由于它创建一个尤伯杯罐子将包含所有你的依赖。
I am trying to run example from learning spark book
package com.example import org.apache.spark.SparkConf import org.apache.spark.streaming.dstream.DStream import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark._ import org.apache.spark.SparkContext._ import org.apache.spark.streaming._ import org.apache.spark.streaming.dstream._ object SimpleExample { def main(args: Array[String]) { val master = args(0) val conf = new SparkConf().setMaster(master).setAppName("StreamingLogInput") // Create a StreamingContext with a 1 second batch size val ssc = new StreamingContext(conf, Seconds(1)) // Create a DStream from all the input on port 7777 val lines = ssc.socketTextStream("localhost", 7777) val errorLines = processLines(lines) // Print out the lines with errors, which causes this DStream to be evaluated errorLines.print() // start our streaming context and wait for it to "finish" ssc.start() // Wait for 10 seconds then exit. To run forever call without a timeout ssc.awaitTerminationOrTimeout(10000) ssc.stop() } def processLines(lines: DStream[String]) = { // Filter our DStream for lines with "error" lines.filter(_.contains("error")) } }
using maven project this is my pom.xml
<project> <groupId>com.streaming.example</groupId> <artifactId>streaming-example</artifactId> <modelVersion>4.0.0</modelVersion> <name>example</name> <packaging>jar</packaging> <version>0.0.1</version> <dependencies> <dependency> <!-- Spark dependency --> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.1.0</version> <scope>provided</scope> </dependency> <dependency> <!-- Spark dependency --> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>1.2.0</version> </dependency> </dependencies> <properties> <java.version>1.7</java.version> </properties> <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.6</source> <target>1.6</target> </configuration> </plugin> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.1.6</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> <configuration> <args> <!-- work-around for https://issues.scala-lang.org/browse/SI-8358 --> <arg>-nobootcp</arg> </args> </configuration> </plugin> </plugins> </pluginManagement> </build> </project>
but when running it like this :
> spark-submit --class src/scala/com/example/SimpleExample.scala \ > target/streaming-example-0.0.1.jar local[4]
I am getting this error :
java.lang.ClassNotFoundException: src/scala/com/example/SimpleExample at java.lang.Class.forName0(Native Method)
EDIT jar content :
$ jar tf target/streaming-example-0.0.1.jar META-INF/ META-INF/MANIFEST.MF META-INF/maven/ META-INF/maven/com.streaming.example/ META-INF/maven/com.streaming.example/streaming-example/ META-INF/maven/com.streaming.example/streaming-example/pom.xml
解决方案Your SimpleExample.class isn't in your jar.
Check your maven build plugins.
You may consider using the assembly plugin and compiling with :
mvn assembly:assembly
As it create an uber-jar the will contain all your dependencies.
这篇关于试图运行样品应用火花提交,但得到找不到类错误时抛出的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!