java.lang.NoClassDefFoundError的：组织/阿帕奇/火花/流/微博/ TwitterUtils $，同时运行TwitterPopularTags [英] java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags

问题描述

我在星火流和Scala初学者。对于项目的要求，我试图在github上运行TwitterPopularTags例如present。由于SBT装配不是为我工作，我不熟悉SBT我试图使用Maven建筑。很多最初的打嗝之后，我能够创造的jar文件。但是，当试图执行它我收到以下错误。任何人可以帮助我解决这个？

I am a beginner in Spark streaming and Scala. For a project requirement I was trying to run TwitterPopularTags example present in github. As SBT assembly was not working for me and I was not familiar with SBT I am trying to use Maven for building. After a lot of initial hiccups, I was able to create the jar file. But while trying to execute it I am getting the following error. Can anybody help me in resolving this?

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$
    at TwitterPopularTags$.main(TwitterPopularTags.scala:43)
    at TwitterPopularTags.main(TwitterPopularTags.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:331)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterUtils$
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 9 more

我加入了以下依赖性
火花streaming_2.10：1.1.0
火花core_2.10：1.1.0
星火串流-twitter_2.10：1.1.0

I have added following dependencies Spark-streaming_2.10:1.1.0 Spark-core_2.10:1.1.0 Spark-streaming-twitter_2.10:1.1.0

我甚至尝试了1.2.0火花流媒体，微博，但也给了我同样的错误。

I even tried the 1.2.0 for Spark-streaming-twitter but that also was giving me the same error.

感谢您提前帮助。

问候，
VPV

Regards, vpv

推荐答案

感谢你给你的建议。我可以只使用SBT装配来解决这个问题。以下是详细信息关于我是如何做到这一点。

Thank you for giving your suggestion. I was able to resolve this issue by using SBT assembly only. Following is the details regarding how I did this.

星火 - 已美元的Cloudera VM p $ psent
斯卡拉 - 不知道这是Cloudera的present，如果没有，我们可以安装它
SBT - 这还需要安装。我做了我的本地机器上都安装和传输的瓶子到VM。安装此我用下面的链接

Spark - Already present in Cloudera VM Scala - Not sure if this is present in Cloudera, if not we can install it SBT - This also needs to be installed. I did both the installs on my local machine and transferred the Jar to the VM. For installing this I used the following link

https://gist.github.com/visenger/5496675

1）一旦所有这些被创建。我们必须为我们的项目的父文件​​夹。我创建了一个名为Twitter的文件夹中。

1) Once all these are created. We have to create the parent folder for our project. I created a folder called Twitter.

2）创建具有以下结构的Twitter / src目录/主/ Scala的另一个文件夹，该文件夹的名称TwitterPopularTags.scala中创建的.scala文件。这已经从code这是我们从github上有细微的变化。我不得不改变import语句

2) Create another folder with the following structure Twitter/src/main/scala and created the .scala file in this folder with the name TwitterPopularTags.scala. This has slight changes from the code which we got from the github. I had to change the import statements

import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.SparkContext._
import org.apache.spark.streaming.twitter._
import org.apache.spark.SparkConf

3）在此之后，创建父文件夹下另一个文件夹具有以下名称

3) After this, create another folder under the parent folder with the following name

微博/项目

和创建一个名为assembly.sbt的文件。这对装配插件的路径。下面是该文件的全code present。

and create a file with the name assembly.sbt . This has the path for the assembly plugin. Following is the full code present in the file.

resolvers += Resolver.url("sbt-plugin-releases-scalasbt", url("http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/"))

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")

4）一旦上述两个创建，建立项目（微博）的名称build.sbt的父目录中的文件。这是我们需要提供我们需要创建，也依赖.jar文件的名称。请注意，在这个文件中的codeS之间连空白行是重要的。

4) Once the above two are created, create a file in the parent directory of the project (Twitter) with the name build.sbt. This is where we need to provide the name of the .Jar file we need to create and also the dependencies. Please note that even the blank lines between the codes in this file are important.

name := "TwitterPopularTags"

version := "1.0"

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
   {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
   }
}

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" % "provided"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.0" % "provided"

libraryDependencies += "org.apache.spark" %% "spark-streaming-twitter" % "1.2.0" 

libraryDependencies += "org.twitter4j" % "twitter4j-stream" % "3.0.3" 

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

5）最后，我们必须打开终端并转到项目（Twitter）上的父文件夹。从这里输入以下命令：

5) Finally we have to open the terminal and go to the parent folder of the project (Twitter). From here enter the following command:

sbt assembly

这将下载的依赖，创造我们所需要的jar文件。

This will download the dependencies and create the jar file we need.

6）为了运行我们需要我们的ID下创建一个Twitter的应用程序的程序，并提供身份验证令牌和其他细节。关于如何创建它的详细步骤是present在下面的链接。

6) In order to run the program we need a twitter app created under our ID and provide the auth token and other details. The detailed step on how to create this is present in following link.

<一个href=\"http://ampcamp.berkeley.edu/3/exercises/realtime-processing-with-spark-streaming.html\">http://ampcamp.berkeley.edu/3/exercises/realtime-processing-with-spark-streaming.html

7）一旦我们有上述所有做我们可以使用火花提交命令从VM运行作业。示例命令

7) Once we have all the above done we can use the spark-submit command from VM to run the job. Example command is

./bin/spark-submit \
  --class TwitterPopularTags \
  --master local[4] \
  /path/to/TwitterPopilarTags.jar \
  comsumerkey consumersecret accesstoken accesssecret 

8）该打印输出到控制台，以便监视输出最好是降低频率
通过调整code。

8) This prints the output to the console so to monitor the output it is better to reduce the frequency by adjusting the code.

请让我知道是否需要任何更多的细节。

Please let me know if any more details are required.

感谢和放大器;问候，

VPV

这篇关于java.lang.NoClassDefFoundError的：组织/阿帕奇/火花/流/微博/ TwitterUtils $，同时运行TwitterPopularTags的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！