开发连接到 azure CosmosDB 的 Scala Spark 应用程序 [英] Developing scala spark app that connect to azure CosmosDB
问题描述
我正在开发连接到 cosmosDB 并且无法解决 SBT 中的依赖关系的 Scala Spark 应用程序.每当我包含 org.apache.spark 它与 azure-cosmosdb-spark 冲突,如果我取出 org.apache.spark 我无法让 spark sparkSession 解决.
Im working on developing scala spark app that connect to cosmosDB and can't resolve dependency within SBT. Whenever I include org.apache.spark it conflict with azure-cosmosdb-spark and if I take out org.apache.spark I can't get spark sparkSession to resolve.
我的 SBT 配置:
name := "MyApp"
version := "1.0"``
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.0",
"org.apache.spark" % "spark-sql_2.11" % "2.3.0" ,
"org.apache.spark" % "spark-streaming_2.11" % "2.3.0" ,
"org.apache.spark" % "spark-mllib_2.11" % "2.3.0" ,
"com.microsoft.azure" % "azure-storage" % "2.0.0",
"org.apache.hadoop" % "hadoop-azure" % "2.7.3",
"com.microsoft.azure" % "azure-cosmosdb-spark_2.2.0_2.11" % "1.0.0",
"com.microsoft.azure" % "azure-documentdb" % "1.14.2" ,
"com.microsoft.azure" % "azure-documentdb-rx" % "0.9.0-rc2" ,
"io.reactivex" % "rxjava" % "1.3.0" ,
"io.reactivex" % "rxnetty" % "0.4.20",
"org.json" % "json" % "20140107",
"org.jmockit" % "jmockit" % "1.34" % "test"
)
推荐答案
您应该使用与 azure-cosmosdb-spark 库完全相同的 Spark 版本.从版本号猜测他们使用的是 2.2.0,而您使用的是 2.3.0.所以你可能需要降级一个版本.
You should use the exact same version of Spark as the azure-cosmosdb-spark library. Guessing from the version number they are using 2.2.0, while you are using 2.3.0. So you probably need to downgrade a version.
如果您真的需要 2.3,则需要查看阴影,例如使用 sbt-assembly 插件.
If you really need 2.3 you will need to look into shading, e.g. with the sbt-assembly plugin.
这篇关于开发连接到 azure CosmosDB 的 Scala Spark 应用程序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!