Apache Zeppelin 0.6.1:运行Spark 2.0 Twitter Stream App [英] Apache Zeppelin 0.6.1: Run Spark 2.0 Twitter Stream App

问题描述

我有一个安装了Spark 2.0和Zeppelin 0.6.1的集群.由于TwitterUtils.scala类已从Spark项目移至Apache Bahir，因此我不能再在Zeppelin笔记本中使用TwitterUtils.

I have a cluster with Spark 2.0 and Zeppelin 0.6.1 installed. Since the class TwitterUtils.scala is moved from Spark project to Apache Bahir, I can't use the TwitterUtils in my Zeppelin notebook anymore.

这是我的笔记本摘要:

依赖项加载:

%dep
z.reset
z.load("org.apache.bahir:spark-streaming-twitter_2.11:2.0.0")

DepInterpreter(%dep) deprecated. Remove dependencies and repositories through GUI interpreter menu instead.
DepInterpreter(%dep) deprecated. Load dependency through GUI interpreter menu instead.
res1: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Dependency@4793109a

还有Spark部分:

import org.apache.spark.streaming.twitter
import org.apache.spark.streaming._
import org.apache.spark.storage.StorageLevel
import scala.io.Source
import scala.collection.mutable.HashMap
import java.io.File
import org.apache.log4j.Logger
import org.apache.log4j.Level
import sys.process.stringSeqToProcess
import org.apache.spark.SparkConf

// ********************************* Configures the Oauth Credentials for accessing Twitter ****************************
def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {...}

// ***************************************** Configure Twitter credentials ********************************************
val apiKey = ...
val apiSecret = ...
val accessToken = ...
val accessTokenSecret = ...
configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)

//  ************************************************* The logic itself *************************************************
val ssc = new StreamingContext(sc, Seconds(2))
val tweets = TwitterUtils.createStream(ssc, None)
val twt = tweets.window(Seconds(60))

当我导入依赖项后尝试运行笔记本的Spark部分时，出现以下异常:

When I try to run the Spark part of the notebook after importing the dependency, I get the following exception:

<console>:44: error: object twitter is not a member of package org.apache.spark.streaming
   import org.apache.spark.streaming.twitter

我在这里做错了什么? Bahir文档还使用import org.apache.spark.streaming.twitter._命令，请参见 http://bahir.apache.org/docs/spark/2.0.0/spark-streaming-twitter/

What am I doing wrong here? Bahir documentation also uses the import org.apache.spark.streaming.twitter._ command, see http://bahir.apache.org/docs/spark/2.0.0/spark-streaming-twitter/

推荐答案

好吧，dep并不完全稳定，而且由于无论如何都已弃用它，为什么不使用受支持的方法呢?如果您既不修改Spark也不修改Zeppelin配置文件，则可以将依赖项添加到解释器配置中(为清楚起见，我省略了属性):

Well, dep is not exactly stable and since it is deprecated anyway why not use supported methods? If you don't won't to modify neither Spark nor Zeppelin configuration files you can add dependencies to the interpreter configuration (I omitted properties for clarity):

这篇关于Apache Zeppelin 0.6.1:运行Spark 2.0 Twitter Stream App的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！