错误在运行SBT装配：SBT重复数据删除错误 [英] error while running sbt assembly : sbt deduplication error

问题描述

我现在面临的具体问题，因为在下面的描述后和建议的答案是没有帮助。
<一href=\"http://stackoverflow.com/questions/25144484/sbt-assembly-deduplication-found-error/25147568#25147568\">sbt-assembly:重复数据删除发现错误

I am facing the exact issue as described in the below post and the suggested answer is not helping. sbt-assembly: deduplication found error

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.transaction\orbits\javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.mail.glassfish\orbits\javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.activation\orbits\javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 14 s, completed Sep 9, 2014 5:21:01 PM

我build.sbt文件包含

my build.sbt file contains

name := "Simple"

version := "0.1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.2"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.0.2"

libraryDependencies += "org.apache.spark" %% "spark-streaming-twitter" % "1.0.2"

libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "0.4.2"

libraryDependencies ++= Seq(
    ("org.apache.spark"%%"spark-core"%"1.0.2").
    exclude("org.eclipse.jetty.orbit", "javax.servlet").
    exclude("org.eclipse.jetty.orbit", "javax.transaction").
    exclude("org.eclipse.jetty.orbit", "javax.mail").
    exclude("org.eclipse.jetty.orbit", "javax.activation").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-collections", "commons-collections").
    exclude("com.esotericsoftware.minlog", "minlog")
)

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

    mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
    {
        case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
        case PathList("javax", "transaction", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "mail", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "activation", xs @ _*)     => MergeStrategy.first
        case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
        case "application.conf" => MergeStrategy.concat
        case "unwanted.txt"     => MergeStrategy.discard
        case x => old(x)
        }
    }

如何解决上述问题的任何指针？

Any pointers on how to fix the above issue?

推荐答案

如果您打算从星火运行您的程序，那么我强烈建议添加所有Spark依赖关系提供所以他们将被排除在装配任务。

If you are planning to run your program from Spark, then I strongly recommend to add all Spark dependencies as provided so they will be excluded from assembly task.

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core"              % "1.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming"         % "1.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.0.2" % "provided")

在另一种情况下，你需要从classpath中删除那些 JAR s或适当的行添加到 mergeStrategy ，在你的情况下，将

In the other case, you need to either remove those jars from classpath or to add appropriate lines to mergeStrategy, in your case it would be

case PathList("META-INF", "ECLIPSEF.RSA") => MergeStrategy.first

如果您仍然希望处理星火的依赖地狱， SBT-依赖性的图表插件应该有所帮助。还要注意的是其他星火依赖性，如火花流和火花流，叽叽喳喳大概需要排除指令了。

If you still wish to deal with Spark's dependencies hell, sbt-dependency-graph plugin should help. Also note that other Spark dependencies, like spark-streaming and spark-streaming-twitter probably needs exclude directive too.

这篇关于错误在运行SBT装配：SBT重复数据删除错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！