如何设置hadoop mapreduce / yarn中的VCORES？ [英] How to set the VCORES in hadoop mapreduce/yarn?

问题描述

以下是我的配置：

  ** mapred-site.xml ** 
 map-mb： 4096 opts：-Xmx3072m 
 reduce-mb：8192 opts：-Xmx6144m 
 
 ** yarn-site.xml ** 
资源内存-mb：40GB 
 min分配-mb：1GB 
  

hadoop集群中的Vcores显示8GB，但我不知道计算或其中的位置

$ b

希望有人能帮助我。

如果您刚刚在单节点集群上运行hadoop，或者在单节点集群上运行hadoop，那么这很可能并不重要，或者，即使是小型的个人分布式集群。您只需要担心内存。

长答案 使用vCores对于较大的群集，以限制不同用户或应用程序的CPU。如果你为自己使用YARN，没有任何真正的理由限制你的容器CPU。这就是为什么在Hadoop中默认不考虑vCore的原因！

尝试将您可用的nodemanager vcore设置为1.它并不重要！你的容器数量仍然是2或4 ..或者任何价值：
$ b $ p $ yarn $ node.mode yarn.nodemanager.resource.memory-mb code> / mapreduce。[map | reduce] .memory.mb

如果真的需要

yarn.nodemanager.resource.cpu-vcores / mapreduce。[map | reduce] .cpu.vcores

然后你需要使用不同的资源计算器。转到您的 capacity-scheduler.xml 配置并将 DefaultResourceCalculator 更改为 DominantResourceCalculator 。

除了使用vCores进行容器分配外，您还想使用vCores真正限制每个节点的CPU使用情况？您需要更改更多配置才能使用 LinuxContainerExecutor 而不是 DefaultContainerExecutor ，因为它可以管理用于限制CPU资源的linux cgroup。按照此页面，如果您想了解更多信息。

The following are my configuration :

**mapred-site.xml**
map-mb : 4096 opts:-Xmx3072m
reduce-mb : 8192 opts:-Xmx6144m

**yarn-site.xml**
resource memory-mb : 40GB
min allocation-mb : 1GB

the Vcores in hadoop cluster displayed 8GB but i dont know how the computation or where to configure it.

hope someone could help me.

解决方案

Short Answer

It most probably doesn't matter, if you are just running hadoop out of the box on your single-node-cluster or even a small personal distributed cluster. You just need to worry about memory.

Long Answer

vCores are used for larger clusters in order to limit CPU for different users or applications. If you are using YARN for yourself there is no real reason to limit your container CPU. That is why vCores are not even taken into consideration by default in Hadoop !

Try setting your available nodemanager vcores to 1. It doesn't matter ! Your number of containers will still be 2 or 4 .. or whatever the value of :

yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb

If really do want the number of containers to take vCores into consideration and be limited by :

yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcores

then you need to use a different a different Resource Calculator. Go to your capacity-scheduler.xml config and change DefaultResourceCalculator to DominantResourceCalculator.

In addition to using vCores for container allocation, you want to use vCores to really limit CPU usage of each node ? You need to change even more configurations to use the LinuxContainerExecutor instead of the DefaultContainerExecutor, because it can manage linux cgroups which are used to limit CPU resources. Follow this page if you want more info on this.

这篇关于如何设置hadoop mapreduce / yarn中的VCORES？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！