星火SQL：TwitterUtils流失败原因不明 [英] Spark SQL: TwitterUtils Streaming fails for unknown reason

问题描述

我使用的是最新的星火主，另外，我加载这些罐子：
  - 火花流-twitter_2.10-1.1.0-SNAPSHOT.jar
  - twitter4j核心-4.0.2.jar
  - twitter4j流-4.0.2.jar

I am using the latest Spark master and additionally, I am loading these jars: - spark-streaming-twitter_2.10-1.1.0-SNAPSHOT.jar - twitter4j-core-4.0.2.jar - twitter4j-stream-4.0.2.jar

这是我在shell执行我的简单的测试程序如下所示：

My simple test program that I execute in the shell looks as follows:

import org.apache.spark.streaming._
import org.apache.spark.streaming.twitter._
import org.apache.spark.streaming.StreamingContext._

System.setProperty("twitter4j.oauth.consumerKey", "jXgXF...")
System.setProperty("twitter4j.oauth.consumerSecret", "mWPvQRl1....")
System.setProperty("twitter4j.oauth.accessToken", "26176....")
System.setProperty("twitter4j.oauth.accessTokenSecret", "J8Fcosm4...")

var ssc = new StreamingContext(sc, Seconds(1))
var tweets = TwitterUtils.createStream(ssc, None)
var statuses = tweets.map(_.getText)
statuses.print()

ssc.start()

不过，我不会得到任何的鸣叫。我看到的主要误差

However, I won't get any tweets. The main error I see is

14/08/04 10:52:35 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
    at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
    ....

和则每次迭代：

INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks

我不知道问题出在哪里。
我如何验证我的Twitter凭据正确识别？
可能会有另一个罐子不见了？

I'm not sure where the problem lies. How can I verify that my twitter credentials are correctly recognized? Might there be another jar missing?

推荐答案

的NoSuchMethodError 要经常因为你要问你是否与图书馆和类相同版本的运行你编译。

NoSuchMethodError should always cause you to ask whether you are running with the same versions of libraries and classes that you compiled with.

如果你看一下星火例子模块的pom.xml文件，你会看到它使用twitter4j 3.0.3。你在运行时将不兼容4.0.2与你和那个打破它。

If you look at the pom.xml file for the Spark examples module, you'll see that it uses twitter4j 3.0.3. You're bringing incompatible 4.0.2 with you at runtime and that breaks it.

这篇关于星火SQL：TwitterUtils流失败原因不明的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！