什么时候使用"sbt程序集"?和"sbt编译"sbt包"? [英] When to use "sbt assembly" and "sbt compile && sbt package"?

问题描述

我想知道什么时候应该使用 sbt汇编，什么时候 sbt编译&&sbt包?

我正在使用Intellij IDEA在本地计算机上编写程序，并使用 sbt compile&&sbt包，然后将其上传到集群并使用 spark-submit 运行它.这是最好的方法吗?什么时候应该使用 sbt程序集?

解决方案

我想知道什么时候应该使用 sbt汇编，什么时候 sbt编译&&sbt包?

引用 sbt-assembly 的网站(该网站提供了 sbt组装>):

目标很简单:使用所有依赖项为项目创建一个胖JAR.

build.sbt 中的任何依赖项(未标记为 Provided )将成为最终jar文件(称为uber-jar)的一部分./p>

您要使用该插件的原因至少有两个:

1. 将所有依赖项捆绑在一起放在一个jar文件中，这样更易​​于部署到生产环境(类似于Docker映像)
2. 使用具有Spark 1.6.0中的 spark-submit 的群集的一个Spark版本(例如2.0.0)运行您的应用程序.

您使用 sbt编译&&sbt package 来编译和打包应用程序的类(顺便说一句，它不必使用Spark).该jar文件不包含应用程序的依赖项，因此必须以其他方式提供任何依赖项(并使部署有些麻烦).

NB: sbt编译&&sbt package 只是 sbt compile package 的一个较长的变体，它假定 compile 任务是 package 的依赖项，而> sbt包.

I would like to know when should I use sbt assembly and when sbt compile && sbt package?

I'm coding a program in my local computer using Intellij IDEA and I'm compiling it with sbt compile && sbt package, then I'm going to upload it to a cluster and run it using spark-submit. Is this the best way? When should I use sbt assembly?

解决方案

I would like to know when should I use sbt assembly and when sbt compile && sbt package?

Quoting sbt-assembly's website (which gives sbt assembly):

The goal is simple: Create a fat JAR of your project with all of its dependencies.

Any dependency in build.sbt (that is not marked as Provided) is going to be part of the final jar file that is called an uber-jar.

There are at least two reasons why you'd like to use the plugin:

1. Have all dependencies bundled together in one jar file that is much easier to deploy to production (akin to a Docker image)
2. Run your application with one Spark version, say 2.0.0, with a cluster that has spark-submit from Spark 1.6.0

You use sbt compile && sbt package to compile and package the classes of your application (which by the way does not have to use Spark whatsoever). That jar file does not include the dependencies of your application so any dependencies have to be provided in some other way (and makes deployment a bit tricker).

NB: sbt compile && sbt package is just a longer variant of sbt compile package which given that compile task is a dependency of package is just sbt package.

这篇关于什么时候使用"sbt程序集"?和"sbt编译"sbt包"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！