使用纱线,火花和火花-cassandra连接器时,如何解决番石榴的11.0.2和16.0之间的冲突? [英] How to resolve the conflict between 11.0.2 and 16.0 of guava when using yarn, spark and spark-cassandra-connector?

查看:60
本文介绍了使用纱线,火花和火花-cassandra连接器时,如何解决番石榴的11.0.2和16.0之间的冲突?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我的毛线版本是 hadop-2.4.0.x ,spark是 spark-1.5.1-bin-hadoop2.4 ,spark-cassandra-connector是<当我执行以下命令时,代码> spark-cassandra-connector_2.10-1.5.0-M2 :

my yarn's version is hadop-2.4.0.x, spark is spark-1.5.1-bin-hadoop2.4 and spark-cassandra-connector is spark-cassandra-connector_2.10-1.5.0-M2, when I executed the following command:

bin/spark-shell --driver-class-path $(echo lib/*.jar | sed 's/ /:/g')  --master yarn-client 
--deploy-mode client --conf spark.cassandra.connection.host=192.21.0.209 
--conf spark.cassandra.auth.username=username --conf spark.cassandra.auth.password=password --conf spark.sql.dialect=sql 
--jars lib/guava-16.0.jar,spark-cassandra-connector_2.10-1.5.0-M2.jar,lib/cassandra-driver-core-2.2.0-rc3.jar

启动后,我在提示符下输入以下scala:

After starting, I input the following scala under the prompt:

import org.apache.spark.sql.cassandra.CassandraSQLContext
import org.apache.spark.sql.{DataFrame, SaveMode}
import org.apache.spark.{Logging, SparkConf, SparkContext}
import org.joda.time.{DateTime, Days, LocalDate}
val cc = new CassandraSQLContext(sc)

val rdd: DataFrame = cc.sql("select user_id,tag_models,dmp_province," +
"zp_gender,zp_age,zp_edu,stg_stage,zp_income,type " +
"from user_center.users_test") 

我遇到了经典错误:

Caused by: java.lang.NoSuchMethodError:  
com.google.common.util.concurrent.Futures.withFallback
(Lcom/google/common/util/concurrent/ListenableFuture;
Lcom/google/common/util/concurrent/FutureFallback;
Ljava/util/concurrent/Executor;)
Lcom/google/common/util/concurrent/ListenableFuture;

google stackoverflower 中搜索此错误后,我知道 guava 的不同版本之间的冲突导致了此错误,并找到了 hadoop 2.4 使用 guava-11.0.2 ,但

After search this error in google and stackoverflower, I know that the conflict between the different versions of guava caused this error, and found hadoop 2.4 use guava-11.0.2 but spark-cassandra-connector_2.10-1.5.0-M2 use guava-16.0.1.

如何解决这种错误,我们将不胜感激!

How to resolve this kind of error, any advice will be appreciated!

更新

很抱歉长时间测试!

现在,对于提交火花,我测试了此分辨率

Now, for spark-submit, I tested this resolution Making Hadoop 2.6 + Spark-Cassandra Driver Play Nice Together successfully under my test yarn cluster

推荐答案

在Hadoop配置中,将以下属性添加到您的 hadoop-env.sh

In Hadoop configuration, add the following property to your hadoop-env.sh

HADOOP_USER_CLASSPATH_FIRST=true

在Spark配置中,还有一个属性可以将其设置为true spark.driver.userClassPathFirst ,但是它仍然是实验性的,仅在集群模式下使用(请参阅火花文档).我个人没有尝试过该属性,但是由于它已在文档中介绍,因此我认为值得一提

In Spark configurations there is also a property you could set it to true spark.driver.userClassPathFirst , but it still an experimental and used only in cluster mode (have a look at the spark documentation). Personally I have not tried this property but since it is presented in the documentation I thought it is worth to mention

这篇关于使用纱线,火花和火花-cassandra连接器时,如何解决番石榴的11.0.2和16.0之间的冲突?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆