星火1.5.1 +斯卡拉2.10 +卡夫卡+卡桑德拉= Java.lang.NoSuchMethodError： [英] Spark 1.5.1 + Scala 2.10 + Kafka + Cassandra = Java.lang.NoSuchMethodError:

问题描述

我想卡夫卡+卡桑德拉连接到星火1.5.1。

库的版本：

  scalaVersion：=2.10.6libraryDependencies ++ = SEQ（
  org.apache.spark％火花streaming_2.10％1.5.1
  org.apache.spark％火花流-kafka_2.10％1.5.1
  com.datastax.spark％火花卡桑德拉 -  connector_2.10％1.5.0-M2
）
 

初​​始化，并使用到应用程序：

  VAL sparkConf =新SparkConf（真）
      .setMaster（本地[2]）
      .setAppName（KafkaStreamToCassandraApp）
      .SET（spark.executor.memory，1G也）
      .SET（spark.cores.max，1）
      .SET（spark.cassandra.connection.host，127.0.0.1）
 

创建模式转换为卡桑德拉是这样的：

  CassandraConnector（sparkConf）.withSessionDo {会话=＆GT;
      session.execute（SDROP KEYSPACE IF EXISTS kafka_streaming）
      session.execute（SCREATE KEYSPACE IF NOT EXISTS具有复制kafka_streaming = {'类'：'SimpleStrategy'，'replication_factor'：1}）
      session.execute（SCREATE TABLE IF NOT EXISTS kafka_streaming.wordcount（字文PRIMARY KEY，算COUNTER））
      session.execute（STRUNCATE kafka_streaming.wordcount）
    }
 

此外，当prepared JAR ，创建几个策略：

  assemblyMergeStrategy汇编：= {
  案例PathList（COM，esotericsoftware，XS @ _ *）=＆GT; MergeStrategy.last
  案例PathList（COM，谷歌，XS @ _ *）=＆GT; MergeStrategy.first
  案例PathList（组织，阿帕奇，XS @ _ *）=＆GT; MergeStrategy.last
  案例PathList（IO，网状，XS @ _ *）=＆GT; MergeStrategy.last
  案例PathList（COM，codahale，XS @ _ *）=＆GT; MergeStrategy.last
  案例PathList（META-INF，io.netty.versions.properties）=＆GT; MergeStrategy.first
 

我认为这个问题与连接

 情况下PathList（COM，谷歌，XS @ _ *）=＆GT; MergeStrategy.first
 

捆绑使用 MergeStrategy.last 。

任何想法？

得到了异常：

 异常线程mainjava.lang.NoSuchMethodError：com.google.common.reflect.TypeToken.isPrimitive（）z
        在com.datastax.driver.core.Type codeC＆LT;＆初始化GT;（输入codec.java：142）。
        在com.datastax.driver.core.Type codeC＆LT;＆初始化GT;（输入codec.java：136）。
        在com.datastax.driver.core.Type codeC $斑点codeC＆LT;＆初始化GT;（输入codec.java：609）。
        在com.datastax.driver.core.Type codeC $斑点codeC＆LT; clinit＆GT;（输入codec.java：606）。
        在com.datastax.driver.core codecRegistry＆LT; clinit＆GT;（codecRegistry.java：147）。
        在com.datastax.driver.core.Configuration $ Builder.build（Configuration.java:259）
        在com.datastax.driver.core.Cluster $ Builder.getConfiguration（Cluster.java:1135）
        在com.datastax.driver.core.Cluster＆LT;＆初始化GT;（Cluster.java:111）
        在com.datastax.driver.core.Cluster.buildFrom（Cluster.java:178）
        在com.datastax.driver.core.Cluster $ Builder.build（Cluster.java:1152）
        在com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85)
        在com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
 

解决方案

根据误差

  [错误] /home/user/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.0.jar:com/google/common/base/Optional.class
 [错误] /home/user/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:com/google/common/base/Optional.class
 

似乎最后是最新的一个，也许就可以把：

 情况下PathList（COM，谷歌，普通，基地，XS @ _ *）=＆GT; MergeStrategy.last
 

I want to connect Kafka + Cassandra to the Spark 1.5.1.

The versions of the libraries:

scalaVersion := "2.10.6"

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming_2.10" % "1.5.1",
  "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.1",
  "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.5.0-M2"
)

The initialization and use into app:

   val sparkConf = new SparkConf(true)
      .setMaster("local[2]")
      .setAppName("KafkaStreamToCassandraApp")
      .set("spark.executor.memory", "1g")
      .set("spark.cores.max", "1")
      .set("spark.cassandra.connection.host", "127.0.0.1")

Creates schema into Cassandra like this:

  CassandraConnector(sparkConf).withSessionDo { session =>
      session.execute(s"DROP KEYSPACE IF EXISTS kafka_streaming")
      session.execute(s"CREATE KEYSPACE IF NOT EXISTS kafka_streaming WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1 }")
      session.execute(s"CREATE TABLE IF NOT EXISTS kafka_streaming.wordcount (word TEXT PRIMARY KEY, count COUNTER)")
      session.execute(s"TRUNCATE kafka_streaming.wordcount")
    }

Also when prepared jar, create a few strategies:

assemblyMergeStrategy in assembly := {
  case PathList("com", "esotericsoftware", xs@_*) => MergeStrategy.last
  case PathList("com", "google", xs@_*) => MergeStrategy.first
  case PathList("org", "apache", xs@_*) => MergeStrategy.last
  case PathList("io", "netty", xs@_*) => MergeStrategy.last
  case PathList("com", "codahale", xs@_*) => MergeStrategy.last
  case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first

I think the issue is connected with

  case PathList("com", "google", xs@_*) => MergeStrategy.first

Tied to use MergeStrategy.last.

Any ideas?

Got exception:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.reflect.TypeToken.isPrimitive()Z
        at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142)
        at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136)
        at com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609)
        at com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606)
        at com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147)
        at com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259)
        at com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135)
        at com.datastax.driver.core.Cluster.<init>(Cluster.java:111)
        at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178)
        at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152)
        at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85)
        at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)

解决方案

Based on the error

 [error] /home/user/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.0.jar:com/google/common/base/Optional.class
 [error] /home/user/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:com/google/common/base/Optional.class

It seems the last is the newest one, maybe you can put:

case PathList("com", "google", "common", "base", xs@_*) => MergeStrategy.last

这篇关于星火1.5.1 +斯卡拉2.10 +卡夫卡+卡桑德拉= Java.lang.NoSuchMethodError：的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！