yarn 不尊重 yarn.nodemanager.resource.cpu-vcores [英] yarn is not honouring yarn.nodemanager.resource.cpu-vcores

问题描述

我使用的是 Hadoop-2.4.0，我的系统配置是 24 核，96 GB RAM.

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM.

我正在使用以下配置

mapreduce.map.cpu.vcores=1
yarn.nodemanager.resource.cpu-vcores=10
yarn.scheduler.minimum-allocation-vcores=1
yarn.scheduler.maximum-allocation-vcores=4
yarn.app.mapreduce.am.resource.cpu-vcores=1

yarn.nodemanager.resource.memory-mb=88064
mapreduce.map.memory.mb=3072
mapreduce.map.java.opts=-Xmx2048m

容量调度程序配置

queue.default.capacity=50
queue.default.maximum_capacity=100
yarn.scheduler.capacity.root.default.user-limit-factor=2

使用上述配置，我预计 yarn 每个节点不会启动超过 10 个映射器，但它每个节点启动 28 个映射器.我做错了什么吗??

With above configs, I expect yarn won't launch more than 10 mappers per node, but It is launching 28 mappers per node. Am I doing something wrong??

推荐答案

YARN 运行的容器比分配的内核多，因为默认情况下 DefaultResourceCalculator用来.它只考虑内存.

YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

使用 DominantResourceCalculator，它同时使用 CPU 和内存.

Use DominantResourceCalculator, It uses both cpu and memory.

在 capacity-scheduler.xml 中设置下面的配置

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

更多关于DominantResourceCalculator

这篇关于yarn 不尊重 yarn.nodemanager.resource.cpu-vcores的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！