Spark Streaming - java.lang.NoSuchMethodError 错误 [英] Spark Streaming - java.lang.NoSuchMethodError Error

问题描述

我正在尝试从 Spark Streaming 访问流式推文.

I am trying to access the streaming tweets from Spark Streaming.

这是软件配置.

Ubuntu 14.04.2 LTS

Ubuntu 14.04.2 LTS

scala -version

scala -version

Scala 代码运行器版本 2.11.7 -- 版权所有 2002-2013，LAMP/EPFL

Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

spark-submit --version

spark-submit --version

Spark 1.6.0 版

Spark version 1.6.0

以下是代码.

object PrintTweets
{
 def main(args: Array[String]) {

  // Configure Twitter credentials using twitter.txt
  setupTwitter()

  // Set up a Spark streaming context named "PrintTweets" that runs locally using
  // all CPU cores and one-second batches of data
  val ssc = new StreamingContext("local[*]", "PrintTweets", Seconds(1))

  // Get rid of log spam (should be called after the context is set up)
  setupLogging()

  // Create a DStream from Twitter using our streaming context
  val tweets = TwitterUtils.createStream(ssc, None)


  // Now extract the text of each status update into RDD's using map()
  val statuses = tweets.map(status => status.getText())

  // Print out the first ten
  statuses.print()

  // Kick it all off
  ssc.start()
  ssc.awaitTermination()
 }
}

Utilities.scala

Utilities.scala

object Utilities {
/** Makes sure only ERROR messages get logged to avoid log spam. */
def setupLogging() = {
  import org.apache.log4j.{Level, Logger}   
  val rootLogger = Logger.getRootLogger()
  rootLogger.setLevel(Level.ERROR)   
}

/** Configures Twitter service credentials using twiter.txt in the   main workspace directory */
def setupTwitter() = {
  import scala.io.Source

  for (line <- Source.fromFile("./data/twitter.txt").getLines) {
   val fields = line.split(" ")
   if (fields.length == 2) {
    System.setProperty("twitter4j.oauth." + fields(0), fields(1))
   }
 }

}

}

问题:

因为它需要 twitter4j 库，所以我添加了
eclipse 中的 twitter4j-core-4.0.4、twitter4j-stream-4.0.4 作为外部 jars 构建路径.

Since it needs the twitter4j library, i have added
twitter4j-core-4.0.4, twitter4j-stream-4.0.4 in eclipse build path as external jars.

然后我运行了程序，它没有抛出任何错误.但是推文没有出现在控制台中.它是空的.

Then i ran the program, it didnt throw any error. But the tweets not appearing in console. It were empty.

所以我看到一些论坛并将 twitter4j 降级到 3.0.3.同样在 Eclipse 中，我在 Build Path 窗口中选择了 Scala 2.10 Library 容器.

So i see some forums and downgraded twitter4j to 3.0.3. Also in Eclipse i chosen Scala 2.10 Library container in Build Path window.

在那之后我得到了 java.lang.NoSuchMethodError 运行时错误.

After that i got java.lang.NoSuchMethodError run-time error.

16/05/14 11:46:01 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError:    twitter4j.TwitterStream.addListener(Ltwitter4j/StreamListener;)V
at    org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
at   org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

请帮我解决这个问题.最初我通过使用 Scala 2.11 构建安装了 spark.是不是这个问题.我是否需要卸载所有内容并重新安装 Scala 2.10，然后安装 Spark 预编译包.

Please help me to resolve this. Initially i have installed spark by built using Scala 2.11. Is that the problem. Do i need uninstall everything and re-install Scala 2.10, then Spark pre-compiled package.

或者除了 Scala 2.11，我的系统中是否需要安装 Scala 2.10?

Or apart from Scala 2.11, do i need to have Scala 2.10 in my system?

推荐答案

上述异常似乎是spark version 1.6.0和twitter4j 3.0.3 version不兼容造成的.

twitter4j.TwitterStream 有方法addListener 采用 twitter4j.StreamListener 的实例.

twitter4j.TwitterStream which is being passed in the onStart method of org.apache.spark.streaming.twitter.TwitterReceiver, has method addListener which takes instance of twitter4j.StreamListener.

twitter4j 3.0.3 版本 没有方法 twitter4j.TwitterStream.addListener(StreamListener)，而是有几个其他的 addListener 方法，取StreamListener的子类.

twitter4j 3.0.3 version has no method twitter4j.TwitterStream.addListener(StreamListener), instead it has few other addListener methods, which take the subclass of StreamListener.

twitter4j 4.0.4 version 有所需的方法，所以这就是这个库没有错误的原因.所以改成twitter4j 3.0.3 版本并不能解决问题.

twitter4j 4.0.4 version has the desired method, so that's why no error comes with this library. So changing to twitter4j 3.0.3 version will not solve the problem.

问题出在其他地方.

这篇关于Spark Streaming - java.lang.NoSuchMethodError 错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！