Spark - 为我的 Spark 作业分配了多少 Executor 和 Core [英] Spark - How many Executors and Cores are allocated to my spark job

问题描述

Spark 架构完全围绕执行器和内核的概念展开.我想看看实际上有多少执行程序和内核为我在集群中运行的 Spark 应用程序运行.

Spark architecture is entirely revolves around the concept of executors and cores. I would like to see practically how many executors and cores running for my spark application running in a cluster.

我试图在我的应用程序中使用以下代码段，但没有成功.

I was trying to use below snippet in my application but no luck.

val conf = new SparkConf().setAppName("ExecutorTestJob")
val sc = new SparkContext(conf)
conf.get("spark.executor.instances")
conf.get("spark.executor.cores")

有什么方法可以使用 SparkContext 对象或 SparkConf 对象等获取这些值.

Is there any way to get those values using SparkContext Object or SparkConf object etc..

推荐答案

Scala (Programmatic way) :

getExecutorStorageStatus 和 getExecutorMemoryStatus 都返回包含驱动程序的执行器数量.像下面的示例片段.

Scala (Programmatic way) :

getExecutorStorageStatus and getExecutorMemoryStatus both return the number of executors including driver. like below example snippet.

/** Method that just returns the current active/registered executors
        * excluding the driver.
        * @param sc The spark context to retrieve registered executors.
        * @return a list of executors each in the form of host:port.
        */
       def currentActiveExecutors(sc: SparkContext): Seq[String] = {
         val allExecutors = sc.getExecutorMemoryStatus.map(_._1)
         val driverHost: String = sc.getConf.get("spark.driver.host")
         allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList
       }

sc.getConf.getInt("spark.executor.instances", 1)

类似地获取所有属性并打印如下，您也可以获得内核信息..

similarly get all properties and print like below you may get cores information as well..

sc.getConf.getAll.mkString("
")

或

sc.getConf.toDebugString

主要是 spark.executor.cores 对于执行器 spark.driver.cores 驱动程序应该有这个值.

Mostly spark.executor.cores for executors spark.driver.cores driver should have this value.

以上方法getExecutorStorageStatus和getExecutorMemoryStatus，在python api中未实施

编辑但是可以使用从 SparkSession 公开的 Py4J 绑定访问.

EDIT But can be accessed using Py4J bindings exposed from SparkSession.

sc._jsc.sc().getExecutorMemoryStatus()

这篇关于Spark - 为我的 Spark 作业分配了多少 Executor 和 Core的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！