YARN上的Spark所使用的vcore太少 [英] Spark on YARN too less vcores used

问题描述

我在具有以下设置的YARN群集(HDP 2.4)中使用Spark:

I'm using Spark in a YARN cluster (HDP 2.4) with the following settings:

• 1个主节点
  • 64 GB RAM(可用50 GB)
  • 24核(可用19核)
  • 1 Masternode
    • 64 GB RAM (50 GB usable)
    • 24 cores (19 cores usable)
    每个
    • 64 GB RAM(可用50 GB)
    • 每个24核(可用19核)
    • (一个主机的)所有容器的内存:50 GB
    • 最小容器大小= 2 GB
    • 最大容器大小= 50 GB
    • vcores = 19
    • 最低#vcores/container = 1
    • 最大#vcores/容器= 19

    当我使用命令spark-submit --num-executors 30 --executor-cores 3 --executor-memory 7g --driver-cores 1 --driver-memory 1800m ...运行spark应用程序时，YARN使用以下设置创建31个容器(每个执行器进程+一个驱动程序进程一个):

    When I run my spark application with the command spark-submit --num-executors 30 --executor-cores 3 --executor-memory 7g --driver-cores 1 --driver-memory 1800m ... YARN creates 31 containers (one for each executor process + one driver process) with the following settings:

    • 正确:具有1个核心的主容器&约1800 MB RAM
    • 正确:30个从属容器，每个容器具有〜7 GB RAM
    • 但不正确:根据YARN ResourceManager用户界面，每个从属容器仅运行 1个核心，而不是3个(显示的是95个中的31个，而不是91 = 30 * 3 +1)，请参见下面的屏幕截图
    • Correct: Master container with 1 core & ~1800 MB RAM
    • Correct: 30 slave containers with ~7 GB RAM each
    • BUT INCORRECT: each slave container only runs with 1 core instead of 3, according to the YARN ResourceManager UI (it shows only 31 of 95 in use, instead of 91 = 30 * 3 + 1), see screenshot below

    我的问题在这里:为什么spark-submit参数--executor-cores 3没有作用?

    My question here: Why does the spark-submit parameter --executor-cores 3 have no effect?

    推荐答案

    好吧，似乎与此处讨论的问题相同:

    Ok, seems to be the same issue as discussed here: yarn is not honouring yarn.nodemanager.resource.cpu-vcores The solution also worked for me.

    这篇关于YARN上的Spark所使用的vcore太少的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！