HADOOP YARN-应用程序已添加到调度程序中,尚未激活.集群资源为空时跳过AM分配 [英] HADOOP YARN - Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty
问题描述
我正在为一个项目评估YARN.我试图使简单的分布式外壳程序示例工作.我已经将应用程序提交到提交"阶段,但是它从未启动.这是此行报告的信息:
I am evaluating YARN for a project. I am trying to get the simple distributed shell example to work. I have gotten the application to the SUBMITTED phase, but it never starts. This is the information reported from this line:
ApplicationReport report = yarnClient.getApplicationReport(appId);
Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = DEFAULT_PARTITION; AM Resource Request = memory:1024, vCores:1; Queue Resource Limit for AM = memory:0, vCores:0; User AM Resource Limit of the queue = memory:0, vCores:0; Queue AM Resource Usage = memory:128, vCores:1;
针对其他开发人员的解决方案似乎必须将yarn-site.xml文件中的yarn.scheduler.capacity.maximum-am-resource-percent
从其默认值 .1 增加.我尝试了 .2 和 .5 的值,但这似乎无济于事.
The solutions for other developers seems to have to increase yarn.scheduler.capacity.maximum-am-resource-percent
in the yarn-site.xml file from its default value of .1. I have tried values of .2 and .5 but it does not seem to help.
推荐答案
好像没有正确配置分配给Yarn的RAM.如果您尝试根据自己的安装从教程中进行推断/修改,则这可能是一个问题.我会强烈建议,您使用诸如
Looks like you did not configure the RAM allocated to Yarn in a proper way. This can be a pin in the ..... if you try to infer/adapt from tutorials according to your own installation. I would strongly recommend that you use tools such as this one:
wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
rm hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
mv hdp_manual_install_rpm_helper_files-2.6.0.3.8/ hdp_conf_files
python hdp_conf_files/scripts/yarn-utils.py -c 4 -m 8 -d 1 false
-
每个节点的
-
-c
个核心数量
每个节点(Giga)的 -
-m
内存量
每个节点的 -
-d
磁盘数量 -
-bool
如果安装了HBase,则为"True";如果不是,则为假" -c
number of cores you have for each node-m
amount of memory you have for each node (Giga)-d
number of disk you have for each node-bool
"True" if HBase is installed; "False" if not
这应该给你类似的东西
Using cores=4 memory=8GB disks=1 hbase=True
Profile: cores=4 memory=5120MB reserved=3GB usableMem=5GB disks=1
Num Container=3
Container Ram=1536MB
Used Ram=4GB
Unused Ram=3GB
yarn.scheduler.minimum-allocation-mb=1536
yarn.scheduler.maximum-allocation-mb=4608
yarn.nodemanager.resource.memory-mb=4608
mapreduce.map.memory.mb=1536
mapreduce.map.java.opts=-Xmx1228m
mapreduce.reduce.memory.mb=3072
mapreduce.reduce.java.opts=-Xmx2457m
yarn.app.mapreduce.am.resource.mb=3072
yarn.app.mapreduce.am.command-opts=-Xmx2457m
mapreduce.task.io.sort.mb=614
相应地编辑yarn-site.xml
和 mapred-site.xml
.
nano ~/hadoop/etc/hadoop/yarn-site.xml
nano ~/hadoop/etc/hadoop/mapred-site.xml
此外,您应该在yarn-site.xml
<property>
<name>yarn.acl.enable</name>
<value>0</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>name_of_your_master_node</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
,并且在您的mapred-site.xml
中:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
然后,使用scp
将conf文件上传到每个节点(如果您将ssh密钥上传到每个节点)
Then, upload your conf files to each node using scp
(If you uploaded you ssh keys to each one)
for node in node1 node2 node3; do scp ~/hadoop/etc/hadoop/* $node:/home/hadoop/hadoop/etc/hadoop/; done
然后重新启动纱线
stop-yarn.sh
start-yarn.sh
并检查您是否可以看到您的节点:
and check that you can see your nodes:
hadoop@master-node:~$ yarn node -list
18/06/01 12:51:33 INFO client.RMProxy: Connecting to ResourceManager at master-node/192.168.0.37:8032
Total Nodes:3
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
node3:34683 RUNNING node3:8042 0
node2:36467 RUNNING node2:8042 0
node1:38317 RUNNING node1:8042 0
This might fix the issue (good luck) (additional info)
这篇关于HADOOP YARN-应用程序已添加到调度程序中,尚未激活.集群资源为空时跳过AM分配的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!