SBT 程序集 jar 排除 [英] SBT assembly jar exclusion

问题描述

我正在使用 spark(在 java API 中)并且需要一个可以推送到集群的 jar，但是 jar 本身不应包含 spark.部署作业的应用程序当然应该包括 spark.

Im using spark (in java API) and require a single jar that can be pushed to the cluster, however the jar itself should not include spark. The app that deploys the jobs of course should include spark.

我想要:

1. sbt run - 一切都应该被编译和执行
2. sbt smallAssembly - 创建一个没有火花的罐子
3. sbt 组装 - 创建一个包含所有内容(包括 spark)的 uber jar，以便于部署.

我有 1. 和 3. 工作.关于我如何 2. 的任何想法?我需要在 build.sbt 文件中添加什么代码?

I have 1. and 3. working. Any ideas on how I can 2. ? What code would I need to add to my build.sbt file?

这个问题不仅与 spark 有关，还与我可能希望排除的任何其他依赖项有关.

The question is not relevant only to spark, but any other dependency that I may wish to exclude as well.

推荐答案

%提供"配置

从胖 jar 中排除 jar 的第一个选项是在库依赖项上使用 "provided" 配置.provided"来自Maven的provided范围定义如下:

% "provided" configuration

The first option to exclude a jar from the fat jar is to use "provided" configuration on the library dependency. "provided" comes from Maven's provided scope that's defined as follows:

这很像 compile，但表示您希望 JDK 或容器在运行时提供依赖项.例如，在为 Java 企业版构建 Web 应用程序时，您可以将 Servlet API 和相关 Java EE API 的依赖设置为 provided 范围，因为 Web 容器提供了这些类.此范围仅在编译和测试类路径上可用，并且不可传递.

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

由于您将代码部署到容器(在本例中为 Spark)，与您的评论相反，您可能需要 Scala 标准库和其他库 jar(例如，如果您使用了 Dispatch).这不会影响 run 或 test.

Since you're deploying your code to a container (in this case Spark), contrary to your comment you'd probably need Scala standard library, and other library jars (e.g. Dispatch if you used it). This won't affect run or test.

如果你只想要你的源代码，而不需要 Scala 标准库或其他库依赖项，那将是 packageBin 内置到 sbt 中.这个打包的 jar 可以与您可以使用 sbt-assembly 的 assemblyPackageDependency 制作的仅依赖 jar 组合.

If you just want your source code, and no Scala standard library or other library dependencies, that would be packageBin built into sbt. This packaged jar can be combined with dependency-only jar you can make using sbt-assembly's assemblyPackageDependency.

最后的选择是在汇编中使用excludedJars:

The final option is to use excludedJars in assembly:

excludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter {_.data.getName == "spark-core_2.9.3-0.8.0-incubating.jar"}
}

这篇关于SBT 程序集 jar 排除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！