星火流在EC2上:螺纹&QUOT异常;主" java.lang.ExceptionInInitializerError [英] Spark Streaming on EC2: Exception in thread "main" java.lang.ExceptionInInitializerError
问题描述
我试图运行火花提交。当我在我的机器上本地运行它,它工作正常,但当部署到Amazon EC2上返回下面的错误。
根@ IP-172-31-47-217斌] $ ./spark-submit --master本地[2] --class main.java.Streamer〜/ streaming-项目-1.0-JAR与 - dependencies.jar
异常线程mainjava.lang.ExceptionInInitializerError
在org.apache.spark.streaming.StreamingContext $<&初始化GT;(StreamingContext.scala:728)。
在org.apache.spark.streaming.StreamingContext $< clinit>(StreamingContext.scala)
在org.apache.spark.streaming.StreamingContext<&初始化GT;(StreamingContext.scala:81)。
在main.java.Streamer $。主要(Streamer.scala:24)
在main.java.Streamer.main(Streamer.scala)
在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)
在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
在java.lang.reflect.Method.invoke(Method.java:606)
在org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
在org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1(SparkSubmit.scala:180)
在org.apache.spark.deploy.SparkSubmit $ .submit(SparkSubmit.scala:205)
在org.apache.spark.deploy.SparkSubmit $。主要(SparkSubmit.scala:120)
在org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NoSuchFieldException:引起SHUTDOWN_HOOK_PRIORITY
在java.lang.Class.getField(Class.java:1592)
在org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:220)
在org.apache.spark.util.ShutdownHookManager $ .shutdownHooks $ lzycompute(ShutdownHookManager.scala:50)
在org.apache.spark.util.ShutdownHookManager $ .shutdownHooks(ShutdownHookManager.scala:48)
在org.apache.spark.util.ShutdownHookManager $ .addShutdownHook(ShutdownHookManager.scala:189)
在org.apache.spark.util.ShutdownHookManager $<&初始化GT;(ShutdownHookManager.scala:58)。
在org.apache.spark.util.ShutdownHookManager $< clinit>(ShutdownHookManager.scala)
... 14个
下面是我的pom.xml文件:
<?XML版本=1.0编码=UTF-8&GT?;
<项目>
<&的groupId GT; astiefel< /的groupId>
<&的artifactId GT;流媒体项目< / artifactId的>
< modelVersion> 4.0.0< / modelVersion>
<名称>横批项目和LT; /名称>
<包装和GT;&罐子LT; /包装>
<&版GT; 1.0 LT; /版本>
<性状>
< maven.compiler.source> 1.6 LT; /maven.compiler.source>
< maven.compiler.target> 1.6 LT; /maven.compiler.target>
<&编码GT; UTF-8< /编码>
< scala.tools.version> 2.10 LT; /scala.tools.version>
<! - 把集群的斯卡拉版本 - >
< scala.version> 2.10.4< /scala.version> < /性状>
<依赖和GT;
<&依赖性GT; <! - 星火依赖 - >
<&的groupId GT; org.apache.spark< /的groupId>
<&的artifactId GT;火花core_2.10< / artifactId的>
<&版GT; 1.5.1< /版本>
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.apache.spark< /的groupId>
<&的artifactId GT;火花streaming_2.10< / artifactId的>
<&版GT; 1.5.1< /版本>
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.scala浪< /的groupId>
<&的artifactId GT;斯卡拉编译< / artifactId的>
<&版GT; $ {scala.version}< /版本>
<&范围GT;&编译LT; /&范围GT;
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.apache.spark< /的groupId>
<&的artifactId GT;火花mllib_2.10< / artifactId的>
<&版GT; 1.0.0< /版本>
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.apache.spark< /的groupId>
<&的artifactId GT;火花mllib_2.10< / artifactId的>
<&版GT; 1.3.1< /版本>
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.scalanlp< /的groupId>
<&的artifactId GT; breeze_2.10< / artifactId的>
<&版GT; 0.10 LT; /版本>
< /依赖性>
<&依赖性GT;
<&的groupId GT; org.apache.spark< /的groupId>
<&的artifactId GT;火花sql_2.10< / artifactId的>
<&版GT; 1.4.0< /版本>
< /依赖性>
< /依赖和GT; <库>
<&库GT;
<&ID GT; Cloudera的-回购释放和LT; / ID>
&LT; URL&GT;的https://repository.cloudera.com/artifactory/repo/< / URL&GT;
&LT; /存储库&GT;
&LT; /存储库&GT; &LT;建立&GT;
&LT; sourceDirectory&GT;的src / main / java的&LT; / sourceDirectory&GT;
&LT;&插件GT;
&LT;&插件GT;
&LT;! - 看到http://davidb.github.com/scala-maven-plugin - &GT;
&LT;&的groupId GT; net.alchim31.maven&LT; /的groupId&GT;
&LT;&的artifactId GT;斯卡拉,Maven的插件&LT; / artifactId的&GT;
&LT;! - &LT;&版GT; 3.1.3&LT; /版本&GT; - &GT;
&LT;&执行GT;
&LT;执行与GT;
&LT;目标&GT;
&LT;&目标GT;&编译LT; /目标&GT;
&LT;&目标GT; testCompile&LT; /目标&GT;
&LT; /目标&GT;
&LT;结构&gt;
&LT;&ARGS GT;
&LT;&ARG GT; -make:传递&LT; / ARG&GT;
&LT;&ARG GT; -dependencyfile&LT; / ARG&GT;
&LT; ARG方式&gt; $ {} project.build.directory / scala_dependencies&LT; / ARG&GT;
&LT; / ARGS&GT;
&LT; /结构&gt;
&LT; /执行&GT;
&LT; /处决&GT;
&LT; /插件&GT;
&LT;&插件GT;
&LT;&的groupId GT; org.apache.maven.plugins&LT; /的groupId&GT;
&LT;&的artifactId GT; Maven的万无一失,插件&LT; / artifactId的&GT;
&LT;! - &LT;&版GT;&2.13 LT; /版本&GT; - &GT;
&LT;结构&gt;
&LT; useFile&GT;假LT; / useFile&GT;
&LT; disableXmlReport&GT;真&LT; / disableXmlReport&GT;
&LT;! - 如果你有问题的classpath像NoDefClassError,... - &GT;
&LT; useManifestOnlyJar&GT;假LT; / useManifestOnlyJar&GT;
&所述包括(includes)GT;
&LT;包括方式&gt; ** / *测试*&LT; / - 包括GT;
&LT;包括方式&gt; ** / *套房*&LT; / - 包括GT;
&LT; / - 包括GT;
&LT; /结构&gt;
&LT; /插件&GT; &LT; - !包命令插件 - &GT;
&LT;&插件GT;
&LT;&的artifactId GT; Maven的组装插件&LT; / artifactId的&GT;
&LT;! - &LT;&版GT; 2.4.1&LT; /版本&GT; - &GT;
&LT;结构&gt;
&LT; descriptorRefs&GT;
&LT; descriptorRef&GT; JAR-具有依赖性和LT; / descriptorRef&GT;
&LT; / descriptorRefs&GT;
&LT; /结构&gt;
&LT;&执行GT;
&LT;执行与GT;
&LT;&ID GT;使组装和LT; / ID&GT;
&LT;阶段&gt;包装及LT; /阶段&gt;
&LT;目标&GT;
&LT;&目标GT;单&LT; /目标&GT;
&LT; /目标&GT;
&LT; /执行&GT;
&LT; /处决&GT;
&LT; /插件&GT;
&LT; /插件&GT;
&LT; /构建&GT;
&LT; /项目&GT;
当您启动火花EC2
默认的Hadoop版本是1.2.1。然而,最近的版本星火(至少1.5.1)要求,在Hadoop的2介绍了hadoop.fs.FileSystem类中的 SHUTDOWN_HOOK_PRIORITY
字段+
一个修复来解决这个问题是启动使用Hadoop版本2+的火花集群。参见火花EC2 --help
的可用选项。例如: - Hadoop的主要版本=纱
将安装的Hadoop版本2.4
I am trying to run spark-submit on a jar file that I created. When I run it locally on my machine it works correctly but when deployed onto Amazon EC2 it returns the following error.
root@ip-172-31-47-217 bin]$ ./spark-submit --master local[2] --class main.java.Streamer ~/streaming-project-1.0-jar-with-dependencies.jar
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.streaming.StreamingContext$.<init>(StreamingContext.scala:728)
at org.apache.spark.streaming.StreamingContext$.<clinit>(StreamingContext.scala)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at main.java.Streamer$.main(Streamer.scala:24)
at main.java.Streamer.main(Streamer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoSuchFieldException: SHUTDOWN_HOOK_PRIORITY
at java.lang.Class.getField(Class.java:1592)
at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:220)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:189)
at org.apache.spark.util.ShutdownHookManager$.<init>(ShutdownHookManager.scala:58)
at org.apache.spark.util.ShutdownHookManager$.<clinit>(ShutdownHookManager.scala)
... 14 more
Below is my pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project>
<groupId>astiefel</groupId>
<artifactId>streaming-project</artifactId>
<modelVersion>4.0.0</modelVersion>
<name>Streamer Project</name>
<packaging>jar</packaging>
<version>1.0</version>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.tools.version>2.10</scala.tools.version>
<!-- Put the Scala version of the cluster -->
<scala.version>2.10.4</scala.version>
</properties>
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>${scala.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.10</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.4.0</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>cloudera-repo-releases</id>
<url>https://repository.cloudera.com/artifactory/repo/</url>
</repository>
</repositories>
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<plugins>
<plugin>
<!-- see http://davidb.github.com/scala-maven-plugin -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<!--<version>3.1.3</version>-->
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-make:transitive</arg>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<!--<version>2.13</version>-->
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<!-- If you have classpath issue like NoDefClassError,... -->
<useManifestOnlyJar>false</useManifestOnlyJar>
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
<!-- "package" command plugin -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<!--<version>2.4.1</version>-->
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
When you launch spark-ec2
the default hadoop version is 1.2.1. However, the recent Spark versions (at least 1.5.1) requires the SHUTDOWN_HOOK_PRIORITY
field in the hadoop.fs.FileSystem class that was introduced in Hadoop 2+.
One fix to get around this problem is to start up your spark cluster with Hadoop version 2+. See spark-ec2 --help
for available options. Example: --hadoop-major-version=yarn
will install version 2.4 of Hadoop.
这篇关于星火流在EC2上:螺纹&QUOT异常;主&QUOT; java.lang.ExceptionInInitializerError的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!