从的NoSuchMethodError火花卡桑德拉 - 连接器组装罐子 [英] NoSuchMethodError from spark-cassandra-connector with assembled jar

问题描述

我是相当新的Scala和想建立一个Spark工作。我已经建立了包含DataStax连接器组装成一个胖子罐子ajob。当我尝试执行它失败了 java.lang.NoSuchMethodError 。我已经破获打开JAR，可以看到，DataStax库包括在内。我失去了一些东西明显？是否有一个很好的教程来看看关于这一过程？

感谢

控制台
    $火花提交--class org.bobbrez.CasCountJob ./target/scala-2.11/bobbrez-spark-assembly-0.0.1.jar KS TN
    ...
    异常线程mainjava.lang.NoSuchMethodError：scala.runtime.ObjectRef.zero（）Lscala /运行/ ObjectRef;
        在com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala)
        在com.datastax.spark.connector.cql.CassandraConnector $$ anonfun $ 2.适用（CassandraConnector.scala：148）
        ...

build.sbt

 名称：=soofa火花版本：=0.0.1scalaVersion：=2.11.7//附加库
libraryDependencies + =org.apache.spark%%火花核％1.6.0％规定
libraryDependencies + =com.datastax.spark%%火花卡桑德拉连接器％1.5.0-M3
libraryDependencies + =com.typesafe％配置％1.3.0mergeStrategy装配＆LT;＆LT; =（mergeStrategy汇编）{（旧）=＆GT;
  {
    当m如果m.toLowerCase.endsWith（MANIFEST.MF）=＆GT; MergeStrategy.discard
    当m如果m.startsWith（META-INF）=＆GT; MergeStrategy.discard
    案例PathList（使用javax，servlet的XS @ _ *）=＆GT; MergeStrategy.first
    案例PathList（组织，阿帕奇，XS @ _ *）=＆GT; MergeStrategy.first
    案例PathList（组织，JBoss的XS @ _ *）=＆GT; MergeStrategy.first
    案about.html=＆GT; MergeStrategy.rename
    案reference.conf=＆GT; MergeStrategy.concat
    案例_ =＆GT; MergeStrategy.first
  }
}
 

CasCountJob.scala

包org.bobbrez//星火
进口org.apache.spark {SparkContext，SparkConf}
进口com.datastax.spark.connector._反对CasCountJob {
  私人VAL AppName的=CasCountJob  高清主（参数：数组[字符串]）{
    的println（你好距离世界+ AppName的）    VAL密钥空间= ARGS（0）
    VAL表名= ARGS（1）    的println（KEYSPACE：+密钥空间）
    的println（表：+表名）    //配置和创建斯卡拉星火语境。
    VAL的conf =新SparkConf（真）
                .SET（spark.cassandra.connection.host，主机名）
                .SET（spark.cassandra.auth.username，USERNAME）
                .SET（spark.cassandra.auth.password，密码）
                .setAppName（AppName的）    VAL SC =新SparkContext（CONF）    VAL RDD = sc.cassandraTable（密钥空间，表名）
    的println（表计数：+ rdd.count）    System.exit（0）
  }
}


解决方案

卡桑德拉连接器，用于星火1.6还处于开发和尚未公布。

有关星火集成卡桑德拉你需要，至少以下依赖关系： -

1. 火花卡桑德拉连接器 - 从这里
   
2. 卡桑德拉核心驱动力 - 从这里下载相应版本>


3. 火花卡桑德拉Java库 - 从这里 下载相应版本LI>
   
4. 其他相关的JAR - jodatime jodatime-转换，< A HREF =htt​​p://central.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar相对=nofollow> jsr166

的卡桑德拉库的相应版本和星火映射提到这里

显然，卡珊德拉连接器星火1.5也正在开发中，你可能会看到一些兼容性问题。卡桑德拉连接器的最稳定的版本是Spark 1.4需要以下JAR文件： -

1. 火花卡桑德拉连接器


2. 卡桑德拉核心驱动


3. 火花卡桑德拉Java库


4. 其他相关的JAR - jodatime jodatime-转换，< A HREF =htt​​p://central.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar相对=nofollow> jsr166

不用说，所有这些jar文件应配置并提供给执行者。

I'm fairly new to Scala and am trying to build a Spark job. I've built ajob that contains the DataStax connector and assembled it into a fat jar. When I try to execute it it fails with a java.lang.NoSuchMethodError. I've cracked open the JAR and can see that the DataStax library is included. Am I missing something obvious? Is there a good tutorial to look at regarding this process?

Thanks

console $ spark-submit --class org.bobbrez.CasCountJob ./target/scala-2.11/bobbrez-spark-assembly-0.0.1.jar ks tn ... Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.zero()Lscala/runtime/ObjectRef; at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148) ...

build.sbt

name := "soofa-spark"

version := "0.0.1"

scalaVersion := "2.11.7"

// additional libraries
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0" %     "provided"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M3"
libraryDependencies += "com.typesafe" % "config" % "1.3.0"

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
    case m if m.startsWith("META-INF") => MergeStrategy.discard
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
    case PathList("org", "apache", xs @ _*) => MergeStrategy.first
    case PathList("org", "jboss", xs @ _*) => MergeStrategy.first
    case "about.html"  => MergeStrategy.rename
    case "reference.conf" => MergeStrategy.concat
    case _ => MergeStrategy.first
  }
}

CasCountJob.scala

package org.bobbrez

// Spark
import org.apache.spark.{SparkContext, SparkConf}
import com.datastax.spark.connector._

object CasCountJob {
  private val AppName = "CasCountJob"

  def main(args: Array[String]) {
    println("Hello world from " + AppName)

    val keyspace = args(0)
    val tablename = args(1)

    println("Keyspace: " + keyspace)
    println("Table: " + tablename)

    // Configure and create a Scala Spark Context.
    val conf = new SparkConf(true)
                .set("spark.cassandra.connection.host", "HOSTNAME")
                .set("spark.cassandra.auth.username",  "USERNAME")
                .set("spark.cassandra.auth.password",  "PASSWORD")
                .setAppName(AppName)

    val sc = new SparkContext(conf)

    val rdd = sc.cassandraTable(keyspace, tablename)
    println("Table Count: " + rdd.count)

    System.exit(0)
  }
}

解决方案

Cassandra connector for Spark 1.6 is still in development and not released yet.

For Integrating Cassandra with Spark you need at-least following dependencies: -

1. Spark-Cassandra connector - Download appropriate version from here
2. Cassandra Core driver - Download appropriate version from here
3. Spark-Cassandra Java library - Download appropriate version from here
4. Other Dependent Jars - jodatime , jodatime-convert, jsr166

The mapping of appropriate version of Cassandra Libraries and Spark are mentioned here

Apparently the Cassandra connector for Spark 1.5 is also is in development and you may see some compatibility issues. The most stable release of Cassandra connector is for Spark 1.4 which requires following Jar Files: -

1. Spark-Cassandra connector
2. Cassandra Core driver
3. Spark-Cassandra Java library
4. Other Dependent Jars - jodatime , jodatime-convert, jsr166

Needless to mention that all these jar files should be configured and available to executors.

这篇关于从的NoSuchMethodError火花卡桑德拉 - 连接器组装罐子的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！