纱线不遵守yarn.nodemanager.resource.cpu-vcores [英] yarn is not honouring yarn.nodemanager.resource.cpu-vcores

问题描述

我正在使用 Hadoop-2.4.0 ，我的系统配置是24核心，96 GB内存。

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM.

我正在使用以下配置：

I am using following configs

mapreduce.map.cpu.vcores=1
yarn.nodemanager.resource.cpu-vcores=10
yarn.scheduler.minimum-allocation-vcores=1
yarn.scheduler.maximum-allocation-vcores=4
yarn.app.mapreduce.am.resource.cpu-vcores=1

yarn.nodemanager.resource.memory-mb=88064
mapreduce.map.memory.mb=3072
mapreduce.map.java.opts=-Xmx2048m

容量调度程序配置

Capacity Scheduler configs

queue.default.capacity=50
queue.default.maximum_capacity=100
yarn.scheduler.capacity.root.default.user-limit-factor=2

通过上述配置，我预计纱线每个节点不会启动10个以上的映射器，但它将为每个节点启动28个映射器。
我做错了什么？

With above configs, I expect yarn won't launch more than 10 mappers per node, but It is launching 28 mappers per node. Am I doing something wrong??

推荐答案

YARN运行的容器多于分配的内核，因为默认 DefaultResourceCalculator 。它只考虑内存。

YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

使用DominantResourceCalculator，它同时使用cpu和内存。

Use DominantResourceCalculator, It uses both cpu and memory.

在capacity-scheduler.xml中的config下面设置
$ b

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

更多关于 DominantResourceCalculator

这篇关于纱线不遵守yarn.nodemanager.resource.cpu-vcores的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！