如何使用 SBT、Spark 和“提供"高效工作依赖? [英] How to work efficiently with SBT, Spark and "provided" dependencies?

问题描述

我正在使用 Scala 构建一个 Apache Spark 应用程序，并且我正在使用 SBT 来构建它.事情是这样的:

1. 当我在 IntelliJ IDEA 下开发时，我希望将 Spark 依赖项包含在类路径中(我正在启动一个带有主类的常规应用程序)
2. 当我打包应用程序(感谢 sbt-assembly)插件时，我确实不希望 Spark 依赖项包含在我的胖 JAR 中
3. 当我通过 sbt test 运行单元测试时，我希望将 Spark 依赖项包含在类路径中(与 #1 相同，但来自 SBT)

为了匹配约束 #2，我将 Spark 依赖项声明为 provided:

libraryDependencies ++= Seq("org.apache.spark" %% "spark-streaming" % sparkVersion % "提供",...)

然后，

I'm building an Apache Spark application in Scala and I'm using SBT to build it. Here is the thing:

1. when I'm developing under IntelliJ IDEA, I want Spark dependencies to be included in the classpath (I'm launching a regular application with a main class)
2. when I package the application (thanks to the sbt-assembly) plugin, I do not want Spark dependencies to be included in my fat JAR
3. when I run unit tests through sbt test, I want Spark dependencies to be included in the classpath (same as #1 but from the SBT)

To match constraint #2, I'm declaring Spark dependencies as provided:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  ...
)

Then, sbt-assembly's documentation suggests to add the following line to include the dependencies for unit tests (constraint #3):

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

That leaves me with constraint #1 not being full-filled, i.e. I cannot run the application in IntelliJ IDEA as Spark dependencies are not being picked up.

With Maven, I was using a specific profile to build the uber JAR. That way, I was declaring Spark dependencies as regular dependencies for the main profile (IDE and unit tests) while declaring them as provided for the fat JAR packaging. See https://github.com/aseigneurin/kafka-sandbox/blob/master/pom.xml

What is the best way to achieve this with SBT?

解决方案

Use the new 'Include dependencies with "Provided" scope' in an IntelliJ configuration.

这篇关于如何使用 SBT、Spark 和“提供"高效工作依赖?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！