星火应用程序将引发javax.servlet.FilterRegistration [英] Spark application throws javax.servlet.FilterRegistration

问题描述

我使用Scala在本地创建和运行应用程序星火

I'm using Scala to create and run a Spark application locally.

我build.sbt：

My build.sbt:

name := "SparkDemo"

version := "1.0"

scalaVersion := "2.10.4"


libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"    exclude("org.apache.hadoop", "hadoop-client")

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.0"


libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0"  excludeAll(
ExclusionRule(organization = "org.eclipse.jetty"))

libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.0"


libraryDependencies += "org.apache.hbase" % "hbase-client" % "0.98.4-hadoop2"

libraryDependencies += "org.apache.hbase" % "hbase-server" % "0.98.4-hadoop2"

libraryDependencies += "org.apache.hbase" % "hbase-common" % "0.98.4-hadoop2"

mainClass in Compile := Some("demo.TruckEvents")

在运行期间我得到的异常
异常线程mainjava.lang.ExceptionInInitializerError
通话过程中...
java.lang.SecurityException异常：引起类javax.servlet.FilterRegistration的签名者的信息不匹配，在同一个包的其他类的签名者的信息

During runtime I get the exception Exception in thread "main" java.lang.ExceptionInInitializerError during calling of... Caused by: java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package

该例外被触发这里：

val sc = new SparkContext("local", "HBaseTest")

我使用的IntelliJ斯卡拉/ SBT插件。

I am using the IntelliJ Scala/SBT plugin.

我见过，其他人也有这个问题<一href=\"https://ileriseviye.word$p$pss.com/2014/12/04/how-to-fix-class-javax-servlet-filterregistrations-signer-information-does-not-match-signer-information-of-other-classes-in-the-same-package-when-unit-testing-with-spark-streaming/\">suggestion解决方案。但是，这是一个Maven构建....我SBT错在这里？ - 或其他任何建议，我怎么能解决这个问题？
谢谢你。

I've seen that other people have also this problem suggestion solution. But this is a maven build....is my sbt wrong here ? - Or any other suggestion how I can solve this problem ? Thanks.

推荐答案

请参阅我的回答类似的问题<一href=\"http://stackoverflow.com/questions/24463183/hbase-0-96-with-spark-v-1-0/27048490#27048490\">here.在阶级冲突来约，因为HBase的依赖于 org.mortbay.jetty 和星火取决于 org.eclipse.jetty 。我能够从HBase的排除 org.mortbay.jetty 相关性来解决问题。

See my answer to a similar question here. The class conflict comes about because HBase depends on org.mortbay.jetty, and Spark depends on org.eclipse.jetty. I was able to resolve the issue by excluding org.mortbay.jetty dependencies from HBase.

如果您在拉Hadoop的共同，那么你也可能需要从排除的javax.servlet Hadoop的共同。我有一个工作的HBase /星火安装与设置我的SBT的依赖关系如下：

If you're pulling in hadoop-common, then you may also need to exclude javax.servlet from hadoop-common. I have a working HBase/Spark setup with my sbt dependencies set up as follows:

val clouderaVersion = "cdh5.2.0"
val hadoopVersion = s"2.5.0-$clouderaVersion"
val hbaseVersion = s"0.98.6-$clouderaVersion"
val sparkVersion = s"1.1.0-$clouderaVersion"

val hadoopCommon = "org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided" excludeAll ExclusionRule(organization = "javax.servlet")
val hbaseCommon = "org.apache.hbase" % "hbase-common" % hbaseVersion % "provided"
val hbaseClient = "org.apache.hbase" % "hbase-client" % hbaseVersion % "provided"
val hbaseProtocol = "org.apache.hbase" % "hbase-protocol" % hbaseVersion % "provided"
val hbaseHadoop2Compat = "org.apache.hbase" % "hbase-hadoop2-compat" % hbaseVersion % "provided"
val hbaseServer = "org.apache.hbase" % "hbase-server" % hbaseVersion % "provided" excludeAll ExclusionRule(organization = "org.mortbay.jetty")
val sparkCore = "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
val sparkStreaming = "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
val sparkStreamingKafka = "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion exclude("org.apache.spark", "spark-streaming_2.10")

这篇关于星火应用程序将引发javax.servlet.FilterRegistration的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！