每个执行程序容器的Dataproc设置的vcore数 [英] Dataproc set number of vcores per executor container
问题描述
我正在构建一个将在Dataproc上运行的spark应用程序.我计划使用临时集群，并为应用程序的每次执行增加一个新集群.因此，我基本上希望我的工作尽可能多地消耗集群资源，并且我对要求有很好的了解.
我一直在尝试关闭动态分配，并自行设置执行程序实例和核心.目前，我正在使用6个实例和30个核心.
也许这更像是一个毛线问题，但是我发现容器vCores和我的spark executor内核之间的关系有些混乱.在YARN应用程序管理器UI中，我看到生成了7个容器(1个驱动程序和6个执行程序)，每个容器使用1个vCore.但是在spark中，我看到执行者本身正在使用我指定的30个内核.
所以我很好奇执行者是否试图在本质上是1个核心盒子上并行执行30个任务.还是AM gui中显示的vCore是错误的?
如果是前者，则想知道设置此应用程序的最佳方法是什么，因此我最终在每个工作节点上只有一个执行程序，并且使用了所有CPU.
YARN GUI中显示的vCore是错误的.这是一个记录不充分的文件，但是capacity-scheduler的一个已知问题，这是Dataproc的默认问题.值得注意的是，使用Dataproc的默认设置，YARN仅基于内存而不是CPU进行资源箱打包.好处是，此功能对于按工作量按期望的程度超额预订CPU(尤其是在受IO限制的情况下)不同程度的用途更广泛，但是缺点是YARN不会以固定的方式减少CPU的使用量.
有关更改为fair-scheduler的一些讨论，请参见 https://stackoverflow.com/a/43302303/3777211 .以查看YARN中准确表示的vcore分配.但是，对于您而言，这样做可能没有任何好处.使YARN在两个维度上都进行装箱打包更多是一个共享多租户群集"问题，只会使调度问题复杂化.
对于您来说，设置应用程序的最佳方法就是忽略YARN关于vcore的说法；如果每个工作节点只需要一个执行者，则将执行者的内存大小设置为每个节点的YARN中可以容纳的最大值，并使每个执行者的内核数等于每个节点的内核总数.
I'm building a spark application which will run on Dataproc. I plan to use ephemeral clusters, and spin a new one up for each execution of the application. So I basically want my job to eat up as much of the cluster resources as possible, and I have a very good idea of the requirements.
I've been playing around with turning off dynamic allocation and setting up the executor instances and cores myself. Currently I'm using 6 instances and 30 cores a pop.
Perhaps it's more of a yarn question, but I'm finding the relationship between container vCores and my spark executor cores a bit confusing. In the YARN application manager UI I see that 7 containers are spawned (1 driver and 6 executors) and each of these use 1 vCore. Within spark however I see that the executors themselves are using the 30 cores I specified.
So I'm curious if the executors are trying to do 30 tasks in parallel on what is essentially a 1 core box. Or maybe the vCore displayed in the AM gui is erroneous?
If its the former, wondering what the best way is to set this application up so I end up with one executor per worker node, and all the CPUs are used.
The vCore displayed in the YARN GUI is erroneous; this is a not-well-documented but a known issue with the capacity-scheduler, which is Dataproc's default. Notably, with the default settings on Dataproc, YARN is only doing resource bin-packing based on memory rather than CPUs; the benefit is that this is more versatile for oversubscribing CPUs to varying degrees as desired per-workload, especially if something is IO bound, but the downside is that YARN won't be responsible for carving out CPU usage in a fixed manner.
See https://stackoverflow.com/a/43302303/3777211 for some discussion of changing to fair-scheduler to see the vcores allocation accurately represented in YARN. However, in your case there's probably no benefit to doing so; making YARN do bin-packing across both dimensions is more of a "shared multitenant cluster" issue, and only complicates the scheduling problem.
In your case, the best way to set your application up is just to ignore what YARN says about vcores; if you want just one executor per worker node, then set the executor memory size to the maximum that will fit in YARN per node, and make cores per executor equal to the total number of cores per node.
这篇关于每个执行程序容器的Dataproc设置的vcore数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
