如何在 hadoop mapreduce/yarn 中设置 VCORES? [英] How to set the VCORES in hadoop mapreduce/yarn?
问题描述
以下是我的配置:
**mapred-site.xml**
map-mb : 4096 opts:-Xmx3072m
reduce-mb : 8192 opts:-Xmx6144m
**yarn-site.xml**
resource memory-mb : 40GB
min allocation-mb : 1GB
hadoop 集群中的 Vcores 显示 8GB，但我不知道如何计算或在哪里配置它.
the Vcores in hadoop cluster displayed 8GB but i dont know how the computation or where to configure it.
希望有人可以帮助我.
推荐答案
简答
如果您只是在单节点集群甚至小型个人分布式集群上开箱即用地运行 hadoop，这很可能无关紧要.你只需要担心内存.
It most probably doesn't matter, if you are just running hadoop out of the box on your single-node-cluster or even a small personal distributed cluster. You just need to worry about memory.
长答案
vCore 用于较大的集群，以限制不同用户或应用程序的 CPU.如果您自己使用 YARN，则没有真正的理由限制您的容器 CPU.这就是为什么在 Hadoop 中默认情况下甚至不考虑 vCore 的原因！
vCores are used for larger clusters in order to limit CPU for different users or applications. If you are using YARN for yourself there is no real reason to limit your container CPU. That is why vCores are not even taken into consideration by default in Hadoop !
尝试将可用的 nodemanager vcores 设置为 1.没关系！您的容器数量仍将是 2 或 4 .. 或任何值:
Try setting your available nodemanager vcores to 1. It doesn't matter ! Your number of containers will still be 2 or 4 .. or whatever the value of :
yarn.nodemanager.resource.memory-mb/mapreduce.[map|reduce].memory.mb
如果确实希望容器数量考虑 vCore 并受到以下限制:
If really do want the number of containers to take vCores into consideration and be limited by :
yarn.nodemanager.resource.cpu-vcores/mapreduce.[map|reduce].cpu.vcores
那么您需要使用不同的资源计算器.转到您的 capacity-scheduler.xml 配置并将 DefaultResourceCalculator 更改为 DominantResourceCalculator.
then you need to use a different a different Resource Calculator. Go to your capacity-scheduler.xml config and change DefaultResourceCalculator to DominantResourceCalculator.
除了使用 vCores 进行容器分配之外，您还想使用 vCores 来真正限制每个节点的 CPU 使用率吗?您需要更改更多配置以使用 LinuxContainerExecutor 而不是 DefaultContainerExecutor，因为它可以管理用于限制 CPU 资源的 linux cgroup.关注 此页面如果您想了解更多信息.
In addition to using vCores for container allocation, you want to use vCores to really limit CPU usage of each node ? You need to change even more configurations to use the LinuxContainerExecutor instead of the DefaultContainerExecutor, because it can manage linux cgroups which are used to limit CPU resources. Follow this page if you want more info on this.
这篇关于如何在 hadoop mapreduce/yarn 中设置 VCORES?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
