Spark独立数字执行器/核心控制 [英] Spark Standalone Number Executors/Cores Control

问题描述

因此，我有一个带有16核和64GB RAM的Spark独立服务器.我在服务器上同时运行了master和worker.我没有启用动态分配.我正在使用Spark 2.0

So I have a spark standalone server with 16 cores and 64GB of RAM. I have both the master and worker running on the server. I don't have dynamic allocation enabled. I am on Spark 2.0

我不了解的是我提交工作并指定以下内容:

What I dont understand is when I submit my job and specify:

--num-executors 2
--executor-cores 2 

仅应占用4个核心.但是，提交作业时，它将占用全部16个核心，并且无论绕开num-executors参数，都会启动8个执行程序.但是，如果将executor-cores参数更改为4，它将进行相应调整，并且4个执行程序将启动.

Only 4 cores should be taken up. Yet when the job is submitted, it takes all 16 cores and spins up 8 executors regardless, bypassing the num-executors parameter. But if I change the executor-cores parameter to 4 it will adjust accordingly and 4 executors will spin up.

推荐答案

免责声明:我真的不知道--num-executors是否可以在独立模式下工作.我还没看到它在YARN之外使用过.

Disclaimer: I really don't know if --num-executors should work or not in standalone mode. I haven't seen it used outside YARN.

注意:如 Marco 所指出的那样--num-executors

Note: As pointed out by Marco --num-executors is no longer in use on YARN.

您可以通过组合spark.cores.max和spark.executor.cores确定执行者的数量为:

You can effectively control number of executors in standalone mode with static allocation (this works on Mesos as well) by combining spark.cores.max and spark.executor.cores where number of executors is determined as:

floor(spark.cores.max / spark.executor.cores)

例如:

--conf "spark.cores.max=4" --conf "spark.executor.cores=2"

这篇关于Spark独立数字执行器/核心控制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！