异常螺纹＆QUOT;主＆QUOT; java.lang.NoClassDefFoundError的：组织/阿帕奇/火花/ RDD / RDD [英] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD
问题描述
请注意，我比程序员更好dataminer。
我试图从笔者运行书高级分析与星火的例子桑迪Ryza（这些code的例子可以从以下地址下载的 https://github.com/sryza/aas ），
我碰到以下问题。
当我打开Intelij知道这个项目，并尝试运行它，我得到的错误异常线程mainjava.lang.NoClassDefFoundError的：组织/阿帕奇/火花/ RDD / RDD
有谁知道如何解决这个问题呢？结果
这是否意味着我使用的火花版本不对？
Please note that I am better dataminer than programmer.
I am trying to run examples from book "Advanced analytics with Spark" from author Sandy Ryza (these code examples can be downloaded from "https://github.com/sryza/aas"),
and I run into following problem.
When I open this project in Intelij Idea and try to run it, I get error "Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD"
Does anyone know how to solve this issue ?
Does this mean i am using wrong version of spark ?
首先，当我试图运行此code，我得到了错误异常线程mainjava.lang.NoClassDefFoundError的：斯卡拉/产品，但我通过设置斯卡拉-lib目录中的Maven编译解决它。
我使用Maven 3.3.9，Java的1.7.0_79和Scala 2.11.7，火花1.6.1。我想这两个Intelij理念14和15个不同版本的Java（1.7），斯卡拉（2.10）和火花，但都没有成功。
我也使用Windows 7。
我SPARK_HOME和路径变量设置，我可以从命令行执行火花壳。
First when I tried to run this code, I got error "Exception in thread "main" java.lang.NoClassDefFoundError: scala/product", but I solved it by setting scala-lib to compile in maven. I use Maven 3.3.9, Java 1.7.0_79 and scala 2.11.7 , spark 1.6.1. I tried both Intelij Idea 14 and 15 different versions of java (1.7), scala (2.10) and spark, but to no success. I am also using windows 7. My SPARK_HOME and Path variables are set, and i can execute spark-shell from command line.
推荐答案
在这本书中的例子显示出--master参数sparkshell，但你需要为你的环境变量指定适当的。如果您没有安装的Hadoop，你需要在本地启动火花外壳。要执行你可以简单地传递路径，以本地文件的参考样本（文件：///），而不是一个HDFS参考（HDFS：//）
The examples in this book will show a --master argument to sparkshell, but you will need to specify arguments as appropriate for your environment. If you don’t have Hadoop installed you need to start the spark-shell locally. To execute the sample you can simply pass paths to local file reference (file:///), rather than a HDFS reference (hdfs://)
笔者提出一个混合型的发展方式：
The author suggest an hybrid development approach:
请发展的前沿在REPL，正如code片
哈登，将它们移动到一个编译的库。
Keep the frontier of development in the REPL, and, as pieces of code harden, move them over into a compiled library.
因此，样本code被视为编译库，而不是独立的应用程序。您可以通过使用它传递给--jars财产引发壳编译JAR，而Maven是用于编译和管理依赖关系。
Hence the samples code are considered as compiled libraries rather than standalone application. You can make the compiled JAR available to spark-shell by passing it to the --jars property, while maven is used for compiling and managing dependencies.
在书中作者描述了如何simplesparkproject可以执行：
In the book the author describes how the simplesparkproject can be executed:
使用Maven编译和打包项目
use maven to compile and package the project
cd simplesparkproject/
mvn package
与JAR依赖启动火花壳
start the spark-shell with the jar dependencies
spark-shell --master local[2] --driver-memory 2g --jars ../simplesparkproject-0.0.1.jar ../README.md
然后，你可以访问如下您火花壳体内对象：
Then you can access you object within the spark-shell as follows:
val myApp = com.cloudera.datascience.MyApp
不过，如果你想执行示例code作为独立的应用和理念中执行它，你需要修改的pom.xml。
一些依赖的需要进行编译，但在一个火花运行时环境中使用。因此，这些依赖关系都标有在pom.xml提供的范围。
However if you want to execute the sample code as Standalone application and execute it within idea you need to modify the pom.xml. Some of dependencies are required for compilation, but are available in an spark runtime environment. Therefore these dependencies are marked with scope provided in the pom.xml.
<!--<scope>provided</scope>-->
您可以重拍所提供的范围，比你将能够思想的范围内运行的样本。但是你不能再提供这个jar作为火花壳依赖。
you can remake the provided scope, than you will be able to run the samples within idea. But you can not provide this jar as dependency for the spark shell anymore.
请注意：使用maven 3.0.5和Java 7+。我曾与Maven的3.3.X版本的插件版本的问题。
Note: using maven 3.0.5 and Java 7+. I had problems with maven 3.3.X version with the plugin versions.
这篇关于异常螺纹＆QUOT;主＆QUOT; java.lang.NoClassDefFoundError的：组织/阿帕奇/火花/ RDD / RDD的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
