对成纱集群运行的问题星火招聘 [英] issue Running Spark Job on Yarn Cluster

问题描述

我想在运行我的火花工作的 Hadoop的纱的集群模式，我使用下面的命令：

I want to run my spark Job in Hadoop YARN cluster mode, and I am using the following command:

spark-submit --master yarn-cluster 
             --driver-memory 1g 
             --executor-memory 1g
             --executor-cores 1 
             --class com.dc.analysis.jobs.AggregationJob
               sparkanalitic.jar param1 param2 param3

我收到以下错误，请提示怎么回事错了，是命令正确与否。我使用CDH 5.3.1。

I am getting error below, kindly suggest whats going wrong, is the command correct or not. I am using CDH 5.3.1.

Diagnostics: Application application_1424284032717_0066 failed 2 times due 
to AM Container for appattempt_1424284032717_0066_000002 exited with  
exitCode: 15 due to: Exception from container-launch.

Container id: container_1424284032717_0066_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)  

Container exited with a non-zero exit code 15
.Failing this attempt.. Failing the application.
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.hdfs
     start time: 1424699723648
     final status: FAILED
     tracking URL: http://myhostname:8088/cluster/app/application_1424284032717_0066
     user: hdfs

2015-02-23 19:26:04 DEBUG Client - stopping client from cache: org.apache.hadoop.ipc.Client@4085f1ac
2015-02-23 19:26:04 DEBUG Utils - Shutdown hook called
2015-02-23 19:26:05 DEBUG Utils - Shutdown hook called

任何帮助将大大AP preciated。

Any help would be greatly appreciated.

推荐答案

这可能意味着很多东西，对我们来说，我们得到了类似的错误消息，因为不支持Java类的版本，我们通过删除引用固定的问题Java类在我们的项目。

It can mean a lot of things, for us, we get the similar error message because of unsupported Java class version, and we fixed the problem by deleting the referenced Java class in our project.

使用此命令来查看详细的错误信息：

Use this command to see the detailed error message:

yarn logs -application_id application_1424284032717_0066

这篇关于对成纱集群运行的问题星火招聘的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！