在纱线上产生火花，在/0.0.0.0:8032处连接到ResourceManager [英] spark on yarn, Connecting to ResourceManager at /0.0.0.0:8032

问题描述

我正在用Mac开发的机器上编写一个Spark程序. hadoop的版本是2.6，spark的版本是1.6.2. hadoop集群有3个节点，当然所有这些节点都在linux机器中. 我在思想IDE中以独立模式运行spark程序，它可以成功运行.但是现在，我将其更改为yarn-client模式，它无法成功运行，并显示以下消息:

I was writing a spark program in my developing machine, which is a mac. The version of hadoop is 2.6, the version of spark is 1.6.2. The hadoop cluster have 3 nodes, of course all in linux machine. I run the spark program in idea IDE in spark standalone mode, it works successfully. But now, I change it to yarn-client mode, it doesn't work successfully, and gives the message as follows:

...
2017-02-23 11:01:33,725-[HL] INFO main org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
2017-02-23 11:01:34,839-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:35,842-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:36,847-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:37,854-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...

我已经将相应的配置文件添加到项目的资源目录中.如果我将其制成jar包并使用spark-submit运行此程序，那将是可以的.现在，我想在IDE中以yarn-client模式运行此程序，就像spark独立模式一样.我该如何解决这个问题?谢谢.

I have already added corresponding configuration files to the resources directory of the project. If I make it a jar package and use spark-submit to run this program, it will be ok. Now, I want to run this program in IDE as yarn-client mode, just like spark standalone mode. How can I fix this problem? Thanks.

推荐答案

确保在纱线模式下运行时，Spark可以使用YARN配置.将这些文件core-site.xml，hdfs-site.xml和yarn-site.xml文件添加到spark的conf目录中.
另外请确保yarn-site.xml包含资源管理器的地址

Ensure the YARN configurations are available for Spark to use when running in yarn mode. Add these files core-site.xml, hdfs-site.xml and yarn-site.xml files to the conf directory of spark.
Also make sure, the yarn-site.xml contains the address of the resource manager

<property>
   <name>yarn.resourcemanager.address</name>
   <value>resource_manager_ip:8032</value>
</property>

这篇关于在纱线上产生火花，在/0.0.0.0:8032处连接到ResourceManager的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！