同时采用火花壳番石榴版本 [英] Guava version while using spark-shell
本文介绍了同时采用火花壳番石榴版本的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我试图用通过对dataproc火花壳火花卡桑德拉连接器，但我无法连接到我的集群。看来，有一个版本不匹配，因为类路径，包括从别的地方更古老的番石榴版本，甚至当我指定启动时的正确版本。我怀疑这可能是由所有的Hadoop的相关性在默认情况下放入类路径造成的。
反正有火花壳只使用番石榴的正确版本，没有摆脱所有的Hadoop相关dataproc包括罐子？
相关数据：
启动火花外壳，显示了具有番石榴的正确版本是： $火花壳--packages com.datastax.spark：火花卡桑德拉 - connector_2.10：1.5.0-M3
::加载设置:: URL = jar:file:/usr/lib/spark/lib/spark-assembly-1.5.2-hadoop2.7.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.datastax.spark＃火花卡桑德拉 - connector_2.10添加为依赖
::解决依赖:: org.apache.spark＃火花提交父母; 1.0
confs：[默认]
发现com.datastax.spark＃火花卡桑德拉 - connector_2.10; 1.5.0-M3在中央
发现org.apache.cassandra＃卡桑德拉 - clientutil; 2.2.2中心
发现com.datastax.cassandra＃卡桑德拉驱动核心; 3.0.0-alpha4中央
发现io.netty＃网状处理程序; 4.0.27.Final中部
发现io.netty＃网状缓冲; 4.0.27.Final中部
发现io.netty＃网状常见; 4.0.27.Final中部
发现io.netty＃网状运输; 4.0.27.Final中部
发现io.netty＃netty- codeC; 4.0.27.Final中部
发现com.codahale.metrics＃指标核心; 3.0.2在中央
发现org.slf4j＃SLF4J的API; 1.7.5在中央
发现org.apache.commons＃公地lang3; 3.3.2中心
发现com.google.guava＃番石榴; 16.0.1中部
发现org.joda＃乔达-转换; 1.2中部
发现乔达时间＃乔达时间; 2.3中部
发现com.twitter＃jsr166e; 1.1.0在中央
发现org.scala琅＃斯卡拉反省; 2.10.5在中央
::分辨率报告::解决502ms ::文物升10毫秒
::在使用的模块：
com.codahale.metrics＃指标核心; 3.0.2中央在[默认]
com.datastax.cassandra＃卡桑德拉驱动核心;中央在3.0.0-alpha4 [默认]
com.datastax.spark＃火花卡桑德拉 - connector_2.10; 1.5.0-M3中央在[默认]
com.google.guava＃番石榴; 16.0.1中央在[默认]
com.twitter＃jsr166e; 1.1.0中央在[默认]
io.netty＃网状缓冲; 4.0.27.Final中央在[默认]
io.netty＃netty- codeC; 4.0.27.Final中央在[默认]
io.netty＃网状常见; 4.0.27.Final中央在[默认]
io.netty＃网状处理程序; 4.0.27.Final中央在[默认]
io.netty＃网状运输; 4.0.27.Final中央在[默认]
乔达时间＃乔达时间; 2.3中央在[默认]
org.apache.cassandra＃卡桑德拉 - clientutil; 2.2.2中央在[默认]
org.apache.commons＃公地lang3; 3.3.2中央在[默认]
org.joda＃乔达-转换; 1.2从中部的[默认]
org.scala琅＃斯卡拉-反映;从中央2.10.5 [默认]
org.slf4j＃SLF4J的API; 1.7.5中央在[默认]
-------------------------------------------------- -------------------
| |模块||文物|
| CONF |号|搜索| dwnlded |驱逐||号| dwnlded |
-------------------------------------------------- -------------------
|默认| 16 | 0 | 0 | 0 || 16 | 0 |
-------------------------------------------------- -------------------
::检索:: org.apache.spark＃火花提交父
confs：[默认]
0文物复制，16已经恢复（0KB / 12毫秒）
欢迎来到
____ __
/ __ / __ ___ _____ / / __
_ \\ \\ / _ \\ / _`/ __ /'_ /
/ ___ / .__ / \\ _，_ / _ / / _ / \\ _ \\ 1.5.2版
/ _ /使用Scala版本2.10.4（OpenJDK的64位服务器虚拟机，Java的1.8.0_66-内部）
在EX pressions类型，让他们评估。
类型：帮助更多的信息。
17时38分46秒15/12/10 WARN org.apache.spark.metrics.MetricsSystem：使用默认名称DAGScheduler源，因为spark.app.id未设置。
作为SC星火上下文。ivysettings.xml文件不HIVE_HOME或HIVE_CONF_DIR发现，在/ etc /蜂巢/ conf.dist / ivysettings.xml将使用
ivysettings.xml文件不HIVE_HOME或HIVE_CONF_DIR发现，在/ etc /蜂巢/ conf.dist / ivysettings.xml将使用
17时38分54秒15/12/10 WARN org.apache.hadoop.util.Native codeLoader：无法加载原生的Hadoop库平台...使用内置-java类适用
17时38分54秒15/12/10 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory：短路本地读，不能使用的功能，因为libhadoop无法加载。
作为sqlContext SQL上下文。
堆栈跟踪做初始连接时：
java.io.IOException异常：无法打开在{} 10.240.0.7卡桑德拉本地连接：9042
在com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
在com.datastax.spark.connector.cql.CassandraConnector $$ anonfun $ 2.适用（CassandraConnector.scala：148）
在com.datastax.spark.connector.cql.CassandraConnector $$ anonfun $ 2.适用（CassandraConnector.scala：148）
在com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys（RefCountedCache.scala：31）
在com.datastax.spark.connector.cql.RefCountedCache.acquire（RefCountedCache.scala：56）
在com.datastax.spark.connector.cql.CassandraConnector.openSession（CassandraConnector.scala：81）
在com.datastax.spark.connector.cql.CassandraConnector.withSessionDo（CassandraConnector.scala：109）
在com.datastax.spark.connector.cql.CassandraConnector.withClusterDo（CassandraConnector.scala：120）
在com.datastax.spark.connector.cql.Schema $ .fromCassandra（Schema.scala：249）
在com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:51)
在com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
在com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef（CassandraTableScanRDD.scala：59）
在com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:146)
在com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify（CassandraTableScanRDD.scala：59）
在com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:143)
在org.apache.spark.rdd.RDD $$ anonfun $ $分区2.适用（RDD.scala：239）
在org.apache.spark.rdd.RDD $$ anonfun $ $分区2.适用（RDD.scala：237）
在scala.Option.getOrElse（Option.scala：120）
在org.apache.spark.rdd.RDD.partitions（RDD.scala：237）
在org.apache.spark.SparkContext.runJob（SparkContext.scala：1921）
在org.apache.spark.rdd.RDD.count（RDD.scala：1125）
在IWC万国表$ $$ $$ IWC万国表IWC万国表IWC $$ $$ IWC万国表$$ $$ IWC万国表$$ LT＆;＆初始化GT;（小于控制台＆GT;：34）。
在IWC万国表$ $$ $$ IWC万国表IWC万国表IWC $$ $$ IWC万国表$$ $$ IWC万国表＆LT;＆初始化GT;（小于控制台＆GT;：45）。
在IWC万国表$ $$ $$ IWC万国表IWC万国表IWC $$ $$ IWC万国表$$ LT＆;＆初始化GT;（小于控制台＆GT;：47）。
在IWC万国表$ $$ $$ IWC万国表IWC万国表IWC万国表$$ $$ IWC万国表＆LT;＆初始化GT;（小于控制台＆GT;：49）。
在IWC万国表$ $$ $$ IWC万国表IWC万国表IWC $$ LT＆;＆初始化GT;（小于控制台＆GT;：51）。
在IWC万国表$ $$ $$ IWC万国表IWC万国表＆LT;＆初始化GT;（小于控制台＆GT;：53）。
在IWC万国表$ $$ IWC万国表＆LT;＆初始化GT;（小于控制台＆GT;：55）。
在IWC万国表$＆LT;＆初始化GT;（小于控制台＆GT;：57）。
在与下;初始化＆GT;（小于控制台＆GT;：59）
在与下;初始化＆GT;（小于控制台＆GT;：63）。
在与下; clinit＆GT;（小于控制台＆GT）
在与下;初始化方式＆gt;（小于控制台＆GT;：7）
在与下; clinit＆GT;（小于控制台＆GT）
在$打印（小于控制台＆GT;）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
在sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
在java.lang.reflect.Method.invoke（Method.java:497）
在org.apache.spark.repl.SparkIMain $ ReadEvalPrint.call（SparkIMain.scala：1065）
在org.apache.spark.repl.SparkIMain $ Request.loadAndRun（SparkIMain.scala：1340）
1 org.apache.spark.repl.SparkIMain.loadAndRunReq $（SparkIMain.scala：840）
在org.apache.spark.repl.SparkIMain.inter preT（SparkIMain.scala：871）
在org.apache.spark.repl.SparkIMain.inter preT（SparkIMain.scala：819）
在org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pasteCommand(SparkILoop.scala:825)
在org.apache.spark.repl.SparkILoop $$ anonfun $ standardCommands $ 8.apply（SparkILoop.scala：345）
在org.apache.spark.repl.SparkILoop $$ anonfun $ standardCommands $ 8.apply（SparkILoop.scala：345）
在scala.tools.nsc.inter$p$pter.LoopCommands$LoopCommand$$anonfun$nullary$1.apply(LoopCommands.scala:65)
在scala.tools.nsc.inter$p$pter.LoopCommands$LoopCommand$$anonfun$nullary$1.apply(LoopCommands.scala:65)
在scala.tools.nsc.inter preter.LoopCommands $ NullaryCmd.apply（LoopCommands.scala：76）
在org.apache.spark.repl.SparkILoop.command（SparkILoop.scala：809）
1 org.apache.spark.repl.SparkILoop.processLine $（SparkILoop.scala：657）
1 org.apache.spark.repl.SparkILoop.innerLoop $（SparkILoop.scala：665）
在org.apache.spark.repl.SparkILoop.org $阿帕奇$火花$ REPL $ SparkILoop $$环（SparkILoop.scala：670）
在org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
在org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
在org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
在scala.tools.nsc.util.ScalaClassLoader $ .savingContextLoader（ScalaClassLoader.scala：135）
在org.apache.spark.repl.SparkILoop.org $阿帕奇$火花$ REPL $ SparkILoop $$过程（SparkILoop.scala：945）
在org.apache.spark.repl.SparkILoop.process（SparkILoop.scala：1059）
在org.apache.spark.repl.Main $。主要（Main.scala：31）
在org.apache.spark.repl.Main.main（Main.scala）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
在sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
在java.lang.reflect.Method.invoke（Method.java:497）
在org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
在org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1（SparkSubmit.scala：180）
在org.apache.spark.deploy.SparkSubmit $ .submit（SparkSubmit.scala：205）
在org.apache.spark.deploy.SparkSubmit $。主要（SparkSubmit.scala：120）
在org.apache.spark.deploy.SparkSubmit.main（SparkSubmit.scala）
java.lang.NoSuchMethodError：产生的原因： com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/Listenab
leFuture;
在com.datastax.driver.core.Connection.initAsync（Connection.java:178）
在com.datastax.driver.core.Connection $ Factory.open（Connection.java:742）
在com.datastax.driver.core.ControlConnection.tryConnect（ControlConnection.java:240）
在com.datastax.driver.core.ControlConnection.reconnectInternal（ControlConnection.java:187）
在com.datastax.driver.core.ControlConnection.connect（ControlConnection.java:79）
在com.datastax.driver.core.Cluster $ Manager.init（Cluster.java:1393）
在com.datastax.driver.core.Cluster.getMetadata（Cluster.java:402）
在com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
... 70更多
解决方案
不幸的是，Hadoop的番石榴上的依赖11（它没有提到Futures.withFallback法）是的长期存在的问题确实的 Hadoop的2.7.1仍取决于番石榴11 。
星火核心采用番石榴14，一样可以见过这里但是这是由遮光番石榴星火组件内各地的工作：
$罐子TF /usr/lib/spark/lib/spark-assembly.jar | grep的concurrent.Futures
组织/火花项目/番石榴/ UTIL /并行/期货$ 1.class
组织/火花项目/番石榴/ UTIL /并行/期货$ 2.class
组织/火花项目/番石榴/ UTIL /并行/期货$ 3.class
组织/火花项目/番石榴/ UTIL /并行/期货$ 4.class
组织/火花项目/番石榴/ UTIL /并行/期货$ 5.class
组织/火花项目/番石榴/ UTIL /并行/期货$ 6.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ChainingListenableFuture $ 1.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ChainingListenableFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ CombinedFuture $ 1.class
组织/火花项目/番石榴/ UTIL /并行/期货$ CombinedFuture $ 2.class
组织/火花项目/番石榴/ UTIL /并行/期货$ CombinedFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ FallbackFuture $ 1 $ 1.class
组织/火花项目/番石榴/ UTIL /并行/期货$ FallbackFuture $ 1.class
组织/火花项目/番石榴/ UTIL /并行/期货$ FallbackFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ FutureCombiner.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateCancelledFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateFailedCheckedFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateFailedFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateSuccessfulCheckedFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ ImmediateSuccessfulFuture.class
组织/火花项目/番石榴/ UTIL /并行/期货$ MappingCheckedFuture.class
组织/火花项目/番石榴/ UTIL /并行/ Futures.class$ javap的-cp /usr/lib/spark/lib/spark-assembly.jar org.spark，project.guava.util.concurrent.Futures
从Futures.java编译
公共final类org.spark，project.guava.util.concurrent.Futures {
公共静态＆LT; V，X扩展java.lang.Exception的＆GT; org.spark-project.guava.util.concurrent.CheckedFuture＆LT; V，X＆GT; makeChecked（org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT ;, com.google.common.base.Function＆LT; java.lang.Exception的，X＆GT;）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT; immediateFuture（Ⅴ）;
公共静态＆LT; V，X扩展java.lang.Exception的＆GT; org.spark-project.guava.util.concurrent.CheckedFuture＆LT; V，X＆GT; immediateCheckedFuture（Ⅴ）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT; immediateFailedFuture（java.lang.Throwable的）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT; immediateCancelledFuture（）;
公共静态＆LT; V，X扩展java.lang.Exception的＆GT; org.spark-project.guava.util.concurrent.CheckedFuture＆LT; V，X＆GT; immediateFailedCheckedFuture（X）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT; withFallback（org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT ;, org.spark-project.guava.util.concurrent.FutureFallback＆LT ;?延伸V＆GT;）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT; withFallback（org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT ;, org.spark-project.guava.util.concurrent.FutureFallback＆LT ;?延伸V＆GT ;, java.util.concurrent.Executor）;
公共静态＆LT; I，O＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; O＆GT;变换（org.spark-project.guava.util.concurrent.ListenableFuture＆LT;我＆gt;中org.spark-project.guava.util.concurrent.AsyncFunction＆LT ;?超我，扩展O＆GT;？）;
公共静态＆LT; I，O＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; O＆GT;变换（org.spark-project.guava.util.concurrent.ListenableFuture＆LT;我＆gt;中org.spark-project.guava.util.concurrent.AsyncFunction＆LT ;?超我，延伸Ø＆gt;中java.util.concurrent.Executor） ;
公共静态＆LT; I，O＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; O＆GT;变换（org.spark-project.guava.util.concurrent.ListenableFuture＆LT;我＆gt;中com.google.common.base.Function＆LT ;?超我，扩展O＆GT;？）;
公共静态＆LT; I，O＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; O＆GT;变换（org.spark-project.guava.util.concurrent.ListenableFuture＆LT;我＆gt;中com.google.common.base.Function＆LT ;?超我，延伸Ø＆gt;中java.util.concurrent.Executor）;
公共静态＆LT; I，O＆GT; java.util.concurrent.Future＆LT; O＆GT; lazyTransform（java.util.concurrent.Future＆LT;我＆gt;中com.google.common.base.Function＆LT ;?超我，扩展O＆GT;）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT;解引用（org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT;＆GT;）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT;＆的java.util.List LT; V＆GT;＆GT; allAsList（org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT; ...）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT;＆的java.util.List LT; V＆GT;＆GT; allAsList（java.lang.Iterable＆LT ;?延伸org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT;＆GT;）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT;＆的java.util.List LT; V＆GT;＆GT; successfulAsList（org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT; ...）;
公共静态＆LT; V＆GT; org.spark-project.guava.util.concurrent.ListenableFuture＆LT;＆的java.util.List LT; V＆GT;＆GT; successfulAsList（java.lang.Iterable＆LT ;?延伸org.spark-project.guava.util.concurrent.ListenableFuture＆LT ;?延伸V＆GT;＆GT;）;
公共静态＆LT; V＆GT;无效的addCallback（org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT ;, org.spark-project.guava.util.concurrent.FutureCallback＆LT ;?超V＆GT;）;
公共静态＆LT; V＆GT;无效的addCallback（org.spark-project.guava.util.concurrent.ListenableFuture＆LT; V＆GT ;, org.spark-project.guava.util.concurrent.FutureCallback＆LT ;?超V＆GT ;, java.util.concurrent.Executor）;
公共静态＆LT; V，X扩展java.lang.Exception的＆GT; V GET（java.util.concurrent.Future＆LT; V＆GT ;, java.lang.Class中的＆LT; X＆GT;）抛出X;
公共静态＆LT; V，X扩展java.lang.Exception的＆GT; V GET（java.util.concurrent.Future＆LT; V＆GT ;,长，java.util.concurrent.TimeUnit中，java.lang.Class中的＆LT; X＆GT;）抛出X;
公共静态＆LT; V＆GT; V getUnchecked（java.util.concurrent.Future＆LT; V＆GT;）;
静态的 {};
}
您可以按照说明这里的http://arjon.es/2015/10/12/making-hadoop-2-dot-6-plus-spark-cassandra-driver-play-nice-together/也不要在编译期间遮光自己。火花壳提到您可以用 spark.driver.extraClassPath 一些变化脱身<一个href=\"http://stackoverflow.com/questions/33764342/spark-1-5-1-cassandra-connector-1-5-0-m2-cassandra-2-1-scala-2-10-nosuchmeth\">here,虽然碰撞可能会再继续在各个点出现。
I'm trying to use the spark-cassandra-connector via spark-shell on dataproc, however I am unable to connect to my cluster. It appears that there is a version mismatch since the classpath is including a much older guava version from somewhere else, even when I specify the proper version on startup. I suspect this is likely caused by all the Hadoop dependencies put into the classpath by default.
Is there anyway to have spark-shell use only the proper version of guava, without getting rid of all the Hadoop-related dataproc included jars?
Relevant Data:
Starting spark-shell, showing it having the proper version of Guava: $ spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.10:1.5.0-M3
:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.5.2-hadoop2.7.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.datastax.spark#spark-cassandra-connector_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.datastax.spark#spark-cassandra-connector_2.10;1.5.0-M3 in central
found org.apache.cassandra#cassandra-clientutil;2.2.2 in central
found com.datastax.cassandra#cassandra-driver-core;3.0.0-alpha4 in central
found io.netty#netty-handler;4.0.27.Final in central
found io.netty#netty-buffer;4.0.27.Final in central
found io.netty#netty-common;4.0.27.Final in central
found io.netty#netty-transport;4.0.27.Final in central
found io.netty#netty-codec;4.0.27.Final in central
found com.codahale.metrics#metrics-core;3.0.2 in central
found org.slf4j#slf4j-api;1.7.5 in central
found org.apache.commons#commons-lang3;3.3.2 in central
found com.google.guava#guava;16.0.1 in central
found org.joda#joda-convert;1.2 in central
found joda-time#joda-time;2.3 in central
found com.twitter#jsr166e;1.1.0 in central
found org.scala-lang#scala-reflect;2.10.5 in central
:: resolution report :: resolve 502ms :: artifacts dl 10ms
:: modules in use:
com.codahale.metrics#metrics-core;3.0.2 from central in [default]
com.datastax.cassandra#cassandra-driver-core;3.0.0-alpha4 from central in [default]
com.datastax.spark#spark-cassandra-connector_2.10;1.5.0-M3 from central in [default]
com.google.guava#guava;16.0.1 from central in [default]
com.twitter#jsr166e;1.1.0 from central in [default]
io.netty#netty-buffer;4.0.27.Final from central in [default]
io.netty#netty-codec;4.0.27.Final from central in [default]
io.netty#netty-common;4.0.27.Final from central in [default]
io.netty#netty-handler;4.0.27.Final from central in [default]
io.netty#netty-transport;4.0.27.Final from central in [default]
joda-time#joda-time;2.3 from central in [default]
org.apache.cassandra#cassandra-clientutil;2.2.2 from central in [default]
org.apache.commons#commons-lang3;3.3.2 from central in [default]
org.joda#joda-convert;1.2 from central in [default]
org.scala-lang#scala-reflect;2.10.5 from central in [default]
org.slf4j#slf4j-api;1.7.5 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 16 | 0 | 0 | 0 || 16 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 16 already retrieved (0kB/12ms)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.2
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.8.0_66-internal)
Type in expressions to have them evaluated.
Type :help for more information.
15/12/10 17:38:46 WARN org.apache.spark.metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
Spark context available as sc.
ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/conf.dist/ivysettings.xml will be used
ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/conf.dist/ivysettings.xml will be used
15/12/10 17:38:54 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/10 17:38:54 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
SQL context available as sqlContext.
Stack Trace when doing initial connection:
java.io.IOException: Failed to open native connection to Cassandra at {10.240.0.7}:9042
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:249)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:51)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:146)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:143)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:34)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:45)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:47)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:49)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:51)
at $iwC$$iwC$$iwC.<init>(<console>:53)
at $iwC$$iwC.<init>(<console>:55)
at $iwC.<init>(<console>:57)
at <init>(<console>:59)
at .<init>(<console>:63)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pasteCommand(SparkILoop.scala:825)
at org.apache.spark.repl.SparkILoop$$anonfun$standardCommands$8.apply(SparkILoop.scala:345)
at org.apache.spark.repl.SparkILoop$$anonfun$standardCommands$8.apply(SparkILoop.scala:345)
at scala.tools.nsc.interpreter.LoopCommands$LoopCommand$$anonfun$nullary$1.apply(LoopCommands.scala:65)
at scala.tools.nsc.interpreter.LoopCommands$LoopCommand$$anonfun$nullary$1.apply(LoopCommands.scala:65)
at scala.tools.nsc.interpreter.LoopCommands$NullaryCmd.apply(LoopCommands.scala:76)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:809)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/Listenab
leFuture;
at com.datastax.driver.core.Connection.initAsync(Connection.java:178)
at com.datastax.driver.core.Connection$Factory.open(Connection.java:742)
at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:240)
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:187)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1393)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:402)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
... 70 more
解决方案
Unfortunately, Hadoop's dependency on Guava 11 (which doesn't have the Futures.withFallback method mentioned) is a longstanding issue and indeed Hadoop 2.7.1 still depends on Guava 11.
Spark core uses Guava 14, as can be seen here but this is worked around by shading Guava inside the Spark assembly:
$ jar tf /usr/lib/spark/lib/spark-assembly.jar | grep concurrent.Futures
org/spark-project/guava/util/concurrent/Futures$1.class
org/spark-project/guava/util/concurrent/Futures$2.class
org/spark-project/guava/util/concurrent/Futures$3.class
org/spark-project/guava/util/concurrent/Futures$4.class
org/spark-project/guava/util/concurrent/Futures$5.class
org/spark-project/guava/util/concurrent/Futures$6.class
org/spark-project/guava/util/concurrent/Futures$ChainingListenableFuture$1.class
org/spark-project/guava/util/concurrent/Futures$ChainingListenableFuture.class
org/spark-project/guava/util/concurrent/Futures$CombinedFuture$1.class
org/spark-project/guava/util/concurrent/Futures$CombinedFuture$2.class
org/spark-project/guava/util/concurrent/Futures$CombinedFuture.class
org/spark-project/guava/util/concurrent/Futures$FallbackFuture$1$1.class
org/spark-project/guava/util/concurrent/Futures$FallbackFuture$1.class
org/spark-project/guava/util/concurrent/Futures$FallbackFuture.class
org/spark-project/guava/util/concurrent/Futures$FutureCombiner.class
org/spark-project/guava/util/concurrent/Futures$ImmediateCancelledFuture.class
org/spark-project/guava/util/concurrent/Futures$ImmediateFailedCheckedFuture.class
org/spark-project/guava/util/concurrent/Futures$ImmediateFailedFuture.class
org/spark-project/guava/util/concurrent/Futures$ImmediateFuture.class
org/spark-project/guava/util/concurrent/Futures$ImmediateSuccessfulCheckedFuture.class
org/spark-project/guava/util/concurrent/Futures$ImmediateSuccessfulFuture.class
org/spark-project/guava/util/concurrent/Futures$MappingCheckedFuture.class
org/spark-project/guava/util/concurrent/Futures.class
$ javap -cp /usr/lib/spark/lib/spark-assembly.jar org.spark-project.guava.util.concurrent.Futures
Compiled from "Futures.java"
public final class org.spark-project.guava.util.concurrent.Futures {
public static <V, X extends java.lang.Exception> org.spark-project.guava.util.concurrent.CheckedFuture<V, X> makeChecked(org.spark-project.guava.util.concurrent.ListenableFuture<V>, com.google.common.base.Function<java.lang.Exception, X>);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> immediateFuture(V);
public static <V, X extends java.lang.Exception> org.spark-project.guava.util.concurrent.CheckedFuture<V, X> immediateCheckedFuture(V);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> immediateFailedFuture(java.lang.Throwable);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> immediateCancelledFuture();
public static <V, X extends java.lang.Exception> org.spark-project.guava.util.concurrent.CheckedFuture<V, X> immediateFailedCheckedFuture(X);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> withFallback(org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>, org.spark-project.guava.util.concurrent.FutureFallback<? extends V>);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> withFallback(org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>, org.spark-project.guava.util.concurrent.FutureFallback<? extends V>, java.util.concurrent.Executor);
public static <I, O> org.spark-project.guava.util.concurrent.ListenableFuture<O> transform(org.spark-project.guava.util.concurrent.ListenableFuture<I>, org.spark-project.guava.util.concurrent.AsyncFunction<? super I, ? extends O>);
public static <I, O> org.spark-project.guava.util.concurrent.ListenableFuture<O> transform(org.spark-project.guava.util.concurrent.ListenableFuture<I>, org.spark-project.guava.util.concurrent.AsyncFunction<? super I, ? extends O>, java.util.concurrent.Executor);
public static <I, O> org.spark-project.guava.util.concurrent.ListenableFuture<O> transform(org.spark-project.guava.util.concurrent.ListenableFuture<I>, com.google.common.base.Function<? super I, ? extends O>);
public static <I, O> org.spark-project.guava.util.concurrent.ListenableFuture<O> transform(org.spark-project.guava.util.concurrent.ListenableFuture<I>, com.google.common.base.Function<? super I, ? extends O>, java.util.concurrent.Executor);
public static <I, O> java.util.concurrent.Future<O> lazyTransform(java.util.concurrent.Future<I>, com.google.common.base.Function<? super I, ? extends O>);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<V> dereference(org.spark-project.guava.util.concurrent.ListenableFuture<? extends org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>>);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<java.util.List<V>> allAsList(org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>...);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<java.util.List<V>> allAsList(java.lang.Iterable<? extends org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>>);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<java.util.List<V>> successfulAsList(org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>...);
public static <V> org.spark-project.guava.util.concurrent.ListenableFuture<java.util.List<V>> successfulAsList(java.lang.Iterable<? extends org.spark-project.guava.util.concurrent.ListenableFuture<? extends V>>);
public static <V> void addCallback(org.spark-project.guava.util.concurrent.ListenableFuture<V>, org.spark-project.guava.util.concurrent.FutureCallback<? super V>);
public static <V> void addCallback(org.spark-project.guava.util.concurrent.ListenableFuture<V>, org.spark-project.guava.util.concurrent.FutureCallback<? super V>, java.util.concurrent.Executor);
public static <V, X extends java.lang.Exception> V get(java.util.concurrent.Future<V>, java.lang.Class<X>) throws X;
public static <V, X extends java.lang.Exception> V get(java.util.concurrent.Future<V>, long, java.util.concurrent.TimeUnit, java.lang.Class<X>) throws X;
public static <V> V getUnchecked(java.util.concurrent.Future<V>);
static {};
}
You can follow the instructions here http://arjon.es/2015/10/12/making-hadoop-2-dot-6-plus-spark-cassandra-driver-play-nice-together/ to also do shading yourself during compilation. With spark-shell you may be able to get away with some changes in spark.driver.extraClassPath as mentioned here, though collisions may then continue to arise at various points.
这篇关于同时采用火花壳番石榴版本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
