纱，客户端上的火花执行人如不采取执行人核心数配置。 [英] Spark executor on yarn-client does not take executor core count configuration.

问题描述

不管火花执行人核心数量，为执行人纱容器不使用超过1个核心。

解决方案

纱线显示每1遗嘱执行人，不论核心 spark.executor.cores ，因为默认情况下，<一个href=\"https://apache.googlesource.com/hadoop-common/+/e0c9f893b684246feb5b4adbb95a05a436cdb790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java\"相对=nofollow> DefaultResourceCalculator 使用。它认为只有内存。

 公众诠释computeAvailableContainers（资源可用，需要的资源）{
//只考虑内存
返回available.getMemory（）/ required.getMemory（）;
  }
 

使用DominantResourceCalculator，它采用CPU和内存。

设置以下配置在能力scheduler.xml

<$p$p><$c$c>yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

更多关于 DominantResourceCalculator

Irrespective of the spark executor core count, yarn container for the executor does not use more than 1 core.

解决方案

YARN is showing 1 core per executor irrespective of spark.executor.cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator

这篇关于纱，客户端上的火花执行人如不采取执行人核心数配置。的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！