正确的方法，使使用SBT火花发罐 [英] Proper way to make a Spark Fat Jar using SBT

问题描述

我需要一个胖罐子星火，因为我创造Knime一个自定义节点。基本上它里面Knime执行一个自包含的罐子，我以为一个胖罐是产卵当地星火工作的唯一途径。最终，我们将继续提交作业到远程群集，但现在我需要它产卵这样的。

I need a Fat Jar with Spark because I'm creating a custom node for Knime. Basically it's a self-contained jar executed inside Knime and I assume a Fat Jar is the only way to spawn a local Spark Job. Eventually we will go on submitting a job to a remote cluster but for now I need it to spawn this way.

这是说，我用这个做了一个胖罐： https://github.com/sbt/sbt - 装配

That said, I made a Fat Jar using this: https://github.com/sbt/sbt-assembly

我做了一个空的SBT项目，列入星火依赖核和组装罐。我把它添加到我的自定义Knime节点的清单，并试图生成一个简单的工作（pararellize集合，收集并打印）。它开始，但我得到这个错误：

I made an empty sbt project, included Spark-core in the dependencies and assembled the Jar. I added it to the manifest of my custom Knime node and tried to spawn a simple job (pararellize a collection, collect it and print it). It starts but I get this error:

发现钥匙'akka.version'没有配置设置

No configuration setting found for key 'akka.version'

我不知道如何解决它。

I have no idea how to solve it.

编辑：这是我的build.sbt

this is my build.sbt

name := "SparkFatJar"

version := "1.0"

scalaVersion := "2.11.6"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.3.0"
)


libraryDependencies +=  "com.typesafe.akka" %% "akka-actor" % "2.3.8"

assemblyJarName in assembly := "SparkFatJar.jar"

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

我发现这个mergestrategy为Spark某处在互联网上，但现在我无法找到源头。

I've found this mergestrategy for Spark somewhere on the internet but I can't find the source right now.

推荐答案

我认为这个问题是你如何已经设置 assemblyMergeStrategy 。试试这个：

I think the issue is with how you've setup assemblyMergeStrategy. Try this:

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "application.conf"            => MergeStrategy.concat
  case "reference.conf"              => MergeStrategy.concat
  case x =>
    val baseStrategy = (assemblyMergeStrategy in assembly).value
    baseStrategy(x)
}

这篇关于正确的方法，使使用SBT火花发罐的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！