如何有效使用SBT，Spark和“提供的"依赖关系? [英] How to work efficiently with SBT, Spark and "provided" dependencies?

问题描述

我正在Scala中构建一个Apache Spark应用程序，并且正在使用SBT来构建它.这是东西:

I'm building an Apache Spark application in Scala and I'm using SBT to build it. Here is the thing:

1. 在IntelliJ IDEA下进行开发时，我希望Spark依赖项包含在类路径中(我正在使用主类启动常规应用程序)
2. 当我打包应用程序(由于sbt-assembly)插件时，我不希望将Spark依赖项包含在我的胖JAR中.
3. 当我通过sbt test运行单元测试时，我希望Spark依赖项包含在类路径中(与#1相同，但来自SBT)

1. when I'm developing under IntelliJ IDEA, I want Spark dependencies to be included in the classpath (I'm launching a regular application with a main class)
2. when I package the application (thanks to the sbt-assembly) plugin, I do not want Spark dependencies to be included in my fat JAR
3. when I run unit tests through sbt test, I want Spark dependencies to be included in the classpath (same as #1 but from the SBT)

要匹配约束2，我将Spark依赖项声明为provided:

To match constraint #2, I'm declaring Spark dependencies as provided:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  ...
)

然后， sbt-assembly的文档建议添加以下行以包括单元测试的依赖项(约束3) ):

Then, sbt-assembly's documentation suggests to add the following line to include the dependencies for unit tests (constraint #3):

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

这使我无法完全满足约束#1，即我无法在IntelliJ IDEA中运行该应用程序，因为未获取Spark依赖项.

That leaves me with constraint #1 not being full-filled, i.e. I cannot run the application in IntelliJ IDEA as Spark dependencies are not being picked up.

在Maven中，我使用了特定的配置文件来构建uber JAR.这样，我就将Spark依赖关系声明为主要概要文件(IDE和单元测试)的常规依赖关系，同时将它们声明为胖JAR包装的provided.参见 https://github.com/aseigneurin/kafka-sandbox/blob/master/pom.xml

With Maven, I was using a specific profile to build the uber JAR. That way, I was declaring Spark dependencies as regular dependencies for the main profile (IDE and unit tests) while declaring them as provided for the fat JAR packaging. See https://github.com/aseigneurin/kafka-sandbox/blob/master/pom.xml

使用SBT实现此目标的最佳方法是什么?

What is the best way to achieve this with SBT?

推荐答案

在IntelliJ配置中使用新的包含具有提供的"作用域的依赖关系".

Use the new 'Include dependencies with "Provided" scope' in an IntelliJ configuration.

这篇关于如何有效使用SBT，Spark和“提供的"依赖关系?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！