Spark Local模式下的执行器数量 [英] Number of Executors in Spark Local Mode

问题描述

因此，我正在本地模式下运行Spark作业. 我使用以下命令来运行作业

So I am running a spark job in local mode. I use the following command to run the job

spark-submit --master local[*] --driver-memory 256g --class main.scala.mainClass target/scala-2.10/spark_proj-assembly-1.0.jar 0 large.csv 100 outputFolder2 10

我正在具有32核和256GB RAM的计算机上运行它.创建conf时，请使用以下代码

I am running this on a machine with 32 Cores and 256GB RAM. When creating the conf i use the following code

val conf = new SparkConf().setMaster("local[*]").setAppName("My App")

现在我现在处于本地模式，Spark在单个JVM中运行所有内容，但这并不意味着它仅启动一个驱动程序并将其用作执行程序.在我的时间轴中，它显示添加了一个执行程序驱动程序. 当我进入执行程序"页面时，只有一个执行程序被分配了32个内核

Now I now in local mode, Spark runs everything inside a single JVM, but does that mean it launches only one driver and use it as executor as well. In my time line it shows one executor driver added. And when I go the the Executors page, there is just one executor with 32 cores assigned to it

这是默认行为吗?我期望spark将为每个核心启动一个执行程序，而不是仅一个获得所有核心的执行程序.如果有人可以解释这种行为，那就太好了

Is this the default behavior ? I was expecting spark would launch one executor per core instead of just one executor that gets all the core. If some one can explain the behavior, that would be great

推荐答案

这是默认行为吗?

Is this the default behavior?

在本地模式下，如您所说，您的驱动程序+执行程序是在单个JVM进程内创建的.您所看到的不是执行者，而是您的工作具有多少核心的视图.通常，在本地模式下运行时，您应该只在executors视图中看到驱动程序.

In local mode, your driver + executors are, as you've said, created inside a single JVM process. What you see isn't an executor, it is a view of how many cores your job has at its disposable. Usually when running under local mode, you should only be seeing the driver in the executors view.

如果您查看

If you look at the code for LocalSchedulerBackend, you'll see the following comment:

/**
 * Used when running a local version of Spark where the executor, backend, and master all run in
 * the same JVM. It sits behind a [[TaskSchedulerImpl]] and handles launching tasks on a single
 * Executor (created by the [[LocalSchedulerBackend]]) running locally.

我们在同一个JVM实例执行器中有一个可以处理所有任务的执行器.

We have a single, in the same JVM instance executor which handles all tasks.

这篇关于Spark Local模式下的执行器数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！