火花+ SBT-装配：QUOT;重复数据删除：在下面的＆QUOT发现不同的文件的内容; [英] spark + sbt-assembly: "deduplicate: different file contents found in the following"

问题描述

我跑了火花应用程序，并想包的测试类成胖子罐子。什么是奇怪的是我跑SBT集结号成功，但失败了，当我跑了SBT测试：组装。

我试过 SBT组装：包括测试类的，它没有工作我的情况。

SBT版本：0.13.8

build.sbt：

 进口sbtassembly.AssemblyPlugin._名称：=组装测试版本：=1.0scalaVersion：=2.10.5libraryDependencies ++ = SEQ（
  （org.apache.spark％火花core_2.10％1.3.1提供％）
    .exclude（org.mortbay.jetty，servlet的API）。
    排除（公地的BeanUtils，公地的BeanUtils核）。
    排除（公地集合，公地集合）。
    排除（共享记录，的commons-logging）。
    排除（com.esotericsoftware.minlog，minlog）。排除（com.codahale.metrics，指标为核心），
  org.json4s％json4s-jackson_2.10％3.2.10％计提，
  com.google.inject％吉斯％4.0
）Project.inConfig（测试）（assemblySettings）
 

解决方案

您将在装配定义mergeStratey，因为我就是这样做的我下面的火花应用程序。

  mergeStrategy装配＆LT;＆LT; =（mergeStrategy汇编）{（旧）=＆GT;
  {
    案例PathList（使用javax，servlet的XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（使用javax，激活，XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（组织，阿帕奇，XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（COM，谷歌，XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（COM，esotericsoftware，XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（COM，codahale，XS @ _ *）=＆GT; MergeStrategy.last
    案例PathList（COM，哭诉，XS @ _ *）=＆GT; MergeStrategy.last
    案about.html=＆GT; MergeStrategy.rename
    案META-INF / ECLIPSEF.RSA=＆GT; MergeStrategy.last
    案META-INF / mailcap的=＆GT; MergeStrategy.last
    案META-INF / mimetypes.default的=＆GT; MergeStrategy.last
    案plugin.properties=＆GT; MergeStrategy.last
    案log4j.properties=＆GT; MergeStrategy.last
    当X =＆GT;旧的（X）
  }
}
 

I ran spark application and wanna pack the test classes into the fat jar. What is weird is I ran "sbt assembly" successfully, but failed when I ran "sbt test:assembly".

I tried sbt-assembly : including test classes, it didn't work for my case.

SBT version : 0.13.8

build.sbt:

import sbtassembly.AssemblyPlugin._

name := "assembly-test"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies ++= Seq(
  ("org.apache.spark" % "spark-core_2.10" % "1.3.1" % Provided)
    .exclude("org.mortbay.jetty", "servlet-api").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-logging", "commons-logging").
    exclude("com.esotericsoftware.minlog", "minlog").exclude("com.codahale.metrics", "metrics-core"),
  "org.json4s" % "json4s-jackson_2.10" % "3.2.10" % Provided,
  "com.google.inject" % "guice" % "4.0"
)

Project.inConfig(Test)(assemblySettings)

解决方案

You will have to define mergeStratey in assembly, as what I did for my spark app below.

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
    case PathList("com", "google", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
    case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
    case "about.html" => MergeStrategy.rename
    case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
    case "META-INF/mailcap" => MergeStrategy.last
    case "META-INF/mimetypes.default" => MergeStrategy.last
    case "plugin.properties" => MergeStrategy.last
    case "log4j.properties" => MergeStrategy.last
    case x => old(x)
  }
}

这篇关于火花+ SBT-装配：QUOT;重复数据删除：在下面的＆QUOT发现不同的文件的内容;的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！