SBT装配罐排除 [英] SBT assembly jar exclusion

问题描述

我正在使用spark(在Java API中)，并且需要可以将其推送到集群的单个jar，但是jar本身不应该包含spark.当然，部署作业的应用程序应该包含spark.

Im using spark (in java API) and require a single jar that can be pushed to the cluster, however the jar itself should not include spark. The app that deploys the jobs of course should include spark.

我想要:

1. sbt运行-所有内容都应编译并执行
2. sbt smallAssembly-创建一个没有火花的罐子
3. sbt组装-创建包含所有内容(包括spark)的uber jar，以便于部署.

我有1.和3.工作.关于我该如何做的任何想法2.?我需要在我的build.sbt文件中添加什么代码?

I have 1. and 3. working. Any ideas on how I can 2. ? What code would I need to add to my build.sbt file?

这个问题不仅与spark有关，而且与我可能希望排除的其他任何依赖无关.

The question is not relevant only to spark, but any other dependency that I may wish to exclude as well.

推荐答案

％提供"的配置

从胖罐中排除罐的第一个选择是对库依赖项使用"provided"配置. "provided"来自Maven的提供的范围，其定义如下:

% "provided" configuration

The first option to exclude a jar from the fat jar is to use "provided" configuration on the library dependency. "provided" comes from Maven's provided scope that's defined as follows:

这与compile非常相似，但是表明您希望JDK或容器在运行时提供依赖项.例如，在为Java Enterprise Edition构建Web应用程序时，您将对Servlet API和相关Java EE API的依赖关系设置为范围provided，因为Web容器提供了这些类.此作用域仅在编译和测试类路径上可用，并且不可传递.

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

由于您要将代码部署到容器(在本例中为Spark)，因此与您的注释相反，您可能需要Scala标准库和其他库jar(例如，如果使用它则为Dispatch).这不会影响run或test.

Since you're deploying your code to a container (in this case Spark), contrary to your comment you'd probably need Scala standard library, and other library jars (e.g. Dispatch if you used it). This won't affect run or test.

如果您只想要源代码，而没有Scala标准库或其他库依赖项，则可以将packageBin内置到sbt中.该打包的jar可以与您可以使用sbt-assembly的assemblyPackageDependency制作的仅依赖jar结合使用.

If you just want your source code, and no Scala standard library or other library dependencies, that would be packageBin built into sbt. This packaged jar can be combined with dependency-only jar you can make using sbt-assembly's assemblyPackageDependency.

最后一个选择是使用excludedJars in assembly:

excludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter {_.data.getName == "spark-core_2.9.3-0.8.0-incubating.jar"}
}

这篇关于SBT装配罐排除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！