Spark Kafka 0.10 NoSuchMethodError org.apache.kafka.clients.consumer.KafkaConsumer.assign [英] Spark Kafka 0.10 NoSuchMethodError org.apache.kafka.clients.consumer.KafkaConsumer.assign

问题描述

当我尝试将Kafka集成用于结构化流时，我遇到了NoSuchMethodError.当有第一条记录可用时，堆栈跟踪如下所示:造成原因:java.lang.NoSuchMethodError:

I am running into a NoSuchMethodError when I try to use Kafka integration for Structured Streaming. The stack trace looks like this when there is the very first record available: Caused by: java.lang.NoSuchMethodError:

org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/Collection;)V
        at org.apache.spark.sql.kafka010.CachedKafkaConsumer.createConsumer(CachedKafkaConsumer.scala:56)
        at org.apache.spark.sql.kafka010.CachedKafkaConsumer.<init>(CachedKafkaConsumer.scala:45)
        at org.apache.spark.sql.kafka010.CachedKafkaConsumer$.getOrCreate(CachedKafkaConsumer.scala:349)
        at org.apache.spark.sql.kafka010.KafkaSourceRDD$$anon$1.<init>(KafkaSourceRDD.scala:137)
        at org.apache.spark.sql.kafka010.KafkaSourceRDD.compute(KafkaSourceRDD.scala:136)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
...

我的sbt具有以下依赖性:

My sbt has these dependencies:

libraryDependencies ++= Seq(
    scalaTest % Test,
    "org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
    "org.apache.spark" %% "spark-sql" % "2.1.0" % "provided",
    "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.1.0" % "provided",
    "org.apache.hadoop" % "hadoop-client" % "2.6.0" % "provided"
)

，Scala版本为2.11.8.

and Scala version is 2.11.8.

我能够为数据框运行spark应用程序.仅仅是Kafka集成给了我这个错误.结构化流代码只是文档此处:

I am able to run spark applications for dataframe. It's only the Kafka integration giving me this error. The structured streaming code is just the example from documentation here:

  val ds1 = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "data01:9092,data02:9092,data03:9092")
  .option("subscribe", "cluster-topic-01")
  .load()
  val ds2 = ds1.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
  val query = ds2.writeStream
        .format("console")
        .start()
  query.awaitTermination()

我可以从哪里看问题的任何建议?TIA

Any advice where I can look at the problem? TIA

推荐答案

我的猜测是，在运行时类路径中某个正在加载的流浪Kafka客户端而不是一个 spark-sql-kafka-0-10 具有依赖关系.

My guess is that you have a stray Kafka client somewhere in the runtime classpath that is being loaded instead of the one spark-sql-kafka-0-10 has a dependency on.

这篇关于Spark Kafka 0.10 NoSuchMethodError org.apache.kafka.clients.consumer.KafkaConsumer.assign的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！