星火丕例如与纱线集群模式:联想失去了 [英] Spark Pi Example in Cluster mode with Yarn: Association lost

查看:1208
本文介绍了星火丕例如与纱线集群模式:联想失去了的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我在运行分布式集群星火三个虚拟机。我使用的Spark 1.3.0与下面的Hadoop 2.6.0。

I have three virtual machines running as distributed Spark cluster. I am using Spark 1.3.0 with an underlying Hadoop 2.6.0.

如果我运行星火丕例如

/usr/local/spark130/bin/spark-submit 
--class org.apache.spark.examples.SparkPi  
--master yarn-client /usr/local/spark130/examples/target/spark-examples_2.10-1.3.0.jar  10000

我得到这样的警告/错误,并最终建立一个例外:

I get this warning/errors and eventually an exception:

 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/08 12:37:06 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@virtm4:47128] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/04/08 12:37:12 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@virtm4:45975] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/04/08 12:37:13 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!

当我检查容器的日志,我看到它是SIGTERM-ED

When I check the logs of the container I see that it was SIGTERM-ed

15/04/08 12:37:08 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
15/04/08 12:37:08 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
15/04/08 12:37:08 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/04/08 12:37:12 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
15/04/08 12:37:12 INFO yarn.ApplicationMaster: Final app status: UNDEFINED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
15/04/08 12:37:12 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before final status was reported.)

解决方案:
我解决了这个问题。我现在用java7目录,而不是Java8。这种情况被报告为错误,但却遭到拒绝这样 HTTPS://issues.apache。组织/ JIRA /浏览/ SPARK-6388
然而,改变Java版本所做的工作。

SOLUTION: I solved the problem. I use Java7 now instead of Java8. This situation was reported as bug, but it was rejected as such https://issues.apache.org/jira/browse/SPARK-6388 Yet, changing Java version did work.

推荐答案

我以前也遇到过类似的问题,直到我发现这个的问题

I encountered similar problem before, until I find this issue

尽量明确停止SparkContext例如 sc.stop()

Try to stop your SparkContext instance explicitly sc.stop()

这篇关于星火丕例如与纱线集群模式:联想失去了的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆