开发连接到 azure CosmosDB 的 Scala Spark 应用程序 [英] Developing scala spark app that connect to azure CosmosDB

查看:28
本文介绍了开发连接到 azure CosmosDB 的 Scala Spark 应用程序的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在开发连接到 cosmosDB 并且无法解决 SBT 中的依赖关系的 Scala Spark 应用程序.每当我包含 org.apache.spark 它与 azure-cosmosdb-spark 冲突,如果我取出 org.apache.spark 我无法让 spark sparkSession 解决.

Im working on developing scala spark app that connect to cosmosDB and can't resolve dependency within SBT. Whenever I include org.apache.spark it conflict with azure-cosmosdb-spark and if I take out org.apache.spark I can't get spark sparkSession to resolve.

我的 SBT 配置:

name := "MyApp"
version := "1.0"``
scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.0",
"org.apache.spark" % "spark-sql_2.11" % "2.3.0" ,
"org.apache.spark" % "spark-streaming_2.11" % "2.3.0" ,
"org.apache.spark" % "spark-mllib_2.11" % "2.3.0" ,
"com.microsoft.azure" % "azure-storage" % "2.0.0",
"org.apache.hadoop" % "hadoop-azure" % "2.7.3",
"com.microsoft.azure" % "azure-cosmosdb-spark_2.2.0_2.11" % "1.0.0",
"com.microsoft.azure" % "azure-documentdb" % "1.14.2" ,
"com.microsoft.azure" % "azure-documentdb-rx" % "0.9.0-rc2" ,
"io.reactivex" % "rxjava" % "1.3.0" ,
"io.reactivex" % "rxnetty" % "0.4.20",
 "org.json" % "json" % "20140107",
"org.jmockit" % "jmockit" % "1.34" % "test"
)

推荐答案

您应该使用与 azure-cosmosdb-spark 库完全相同的 Spark 版本.从版本号猜测他们使用的是 2.2.0,而您使用的是 2.3.0.所以你可能需要降级一个版本.

You should use the exact same version of Spark as the azure-cosmosdb-spark library. Guessing from the version number they are using 2.2.0, while you are using 2.3.0. So you probably need to downgrade a version.

如果您真的需要 2.3,则需要查看阴影,例如使用 sbt-assembly 插件.

If you really need 2.3 you will need to look into shading, e.g. with the sbt-assembly plugin.

这篇关于开发连接到 azure CosmosDB 的 Scala Spark 应用程序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆