为什么sbt找不到KafkaUtils? [英] Why does sbt fail to find KafkaUtils?
本文介绍了为什么sbt找不到KafkaUtils?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在使用SBT编译的代码(来自Kafka的wordCount)中存在此错误
I have this error in my code (wordCount from Kafka) compiled with SBT
[error] /home/hduser/sbt_project/project1/src/main/scala/sparkKafka.scala:4:35: object kafka is not a member of package org.apache.spark.streaming`
[error] import org.apache.spark.streaming.kafka.KafkaUtils
not found: value KafkaUtils
[error] val lines = KafkaUtils.createStream(ssc, "localhost:2181", "spark-stream ing-consumer-group", Map("customer" -> 2))
文件 build.sbt
包含以下依赖项:
libraryDependencies += "org.apache.spark" % "spark-core_2.12" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.12" % "2.4.0"
如何正确导入 KafkaUtils
?
推荐答案
KafkaUtils
在 org.apache.spark.streaming.kafka010
包中(请注意,名称空间导入包含版本 kafka010
).
KafkaUtils
is in the org.apache.spark.streaming.kafka010
package (note that the namespace for the import includes the version kafka010
).
import org.apache.spark.streaming.kafka010._
// ...
// val streamingContext = ...
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> "localhost:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "use_a_separate_group_id_for_each_stream",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val topics = Array("topicA", "topicB")
val stream = KafkaUtils.createDirectStream[String, String](
streamingContext,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
注意:通常建议使用 Spark结构化改用Kafka进行流式传输..
这篇关于为什么sbt找不到KafkaUtils?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文