为什么YARN上的驱动程序和执行程序的核心数量与请求的数量不同? [英] Why is the number of cores for driver and executors on YARN different from the number requested?
问题描述
我通过以下方式在集群模式下部署Spark作业
I deploy a spark job in the cluster mode with the following
驱动程序核心-1
执行器核心-2
执行者数量-2.
Driver core - 1
Executor cores - 2
Number of executors - 2.
我的理解是，此应用程序应在集群中占据5个核心(4个执行者核心和1个驱动程序核心)，但是我在RM和Spark UI中没有观察到这一点.
My understanding is that this application should occupy 5 cores in the cluster (4 executor cores and 1 driver core) but i dont observe this in the RM and Spark UIs.
- 在资源管理器UI上，我仅看到此应用程序使用的4个内核.
- 即使在Spark UI中(在RM中单击ApplicationMaster URL)，在执行者"选项卡下，驱动程序核心也显示为零.
我想念什么吗?
集群管理器是YARN.
The cluster manager is YARN.
推荐答案
我的理解是，该应用程序应占据集群中的5个核心(4个执行程序核心和1个驱动程序核心)
My understanding is that this application should occupy 5 cores in the cluster (4 executor cores and 1 driver core)
这是YARN中的完美情况，它可以为您管理的CPU提供5个内核.
That's the perfect situation in YARN where it could give you 5 cores off the CPUs it manages.
但是我没有在RM和Spark UI中观察到这一点.
but i dont observe this in the RM and Spark UIs.
由于通常不会发生完美的情况，因此最好拥有尽可能多的YARN内核，这样Spark应用程序就可以启动.
Since the perfect situation does not occur often something it's nice to have as many cores as we could get from YARN so the Spark application could ever start.
Spark可以无限期地等待所请求的内核，但这可能并不总是您喜欢的，是吗?
Spark could just wait indefinitely for the requested cores, but that could not always be to your liking, could it?
这就是为什么YARN上的Spark进行额外检查(也称为minRegisteredRatio)的原因，该检查是应用程序开始执行任务之前所请求的内核的至少 80％.您可以使用spark.scheduler.minRegisteredResourcesRatio Spark属性来控制比率.那可以解释为什么您看到使用的内核少于请求的数量.
That's why Spark on YARN has an extra check (aka minRegisteredRatio) that's the minimum of 80% of cores requested before the application starts executing tasks. You can use spark.scheduler.minRegisteredResourcesRatio Spark property to control the ratio. That would explain why you see less cores in use than requested.
引用 Spark官方文档(突出显示):
spark.scheduler.minRegisteredResourcesRatio
0.8 (对于YARN模式)
0.8 for YARN mode
注册资源的最小比例(注册资源/总预期资源)(资源是yarn模式下的执行者，独立模式下的CPU核心和Mesos粗粒度模式['spark.cores.max'值是Mesos粗粒度模式])在计划开始之前要等待. 指定为0.0到1.0之间的两倍.不管是否已达到最小资源比例，在计划开始之前它将等待的最大时间量由config spark.scheduler.maxRegisteredResourcesWaitingTime控制.
The minimum ratio of registered resources (registered resources / total expected resources) (resources are executors in yarn mode, CPU cores in standalone mode and Mesos coarsed-grained mode ['spark.cores.max' value is total expected resources for Mesos coarse-grained mode] ) to wait for before scheduling begins. Specified as a double between 0.0 and 1.0. Regardless of whether the minimum ratio of resources has been reached, the maximum amount of time it will wait before scheduling begins is controlled by config spark.scheduler.maxRegisteredResourcesWaitingTime.
这篇关于为什么YARN上的驱动程序和执行程序的核心数量与请求的数量不同?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
