星火配置，是SPARK_DRIVER_MEMORY，SPARK_EXECUTOR_MEMORY和SPARK_WORKER_MEMORY的区别？ [英] Spark configuration, what is the difference of SPARK_DRIVER_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_WORKER_MEMORY?

问题描述

我做我的工作，请阅读 https://spark.apache.org文档/docs/latest/configuration.html

I did my work, read the documentation at https://spark.apache.org/docs/latest/configuration.html

在火花文件夹/ conf目录/ spark-env.sh：

in spark-folder/conf/spark-env.sh:

• SPARK_DRIVER_MEMORY ，记忆硕士（例如1000M，2G）（默认值：512 MB）


• SPARK_EXECUTOR_MEMORY ，每个工人（例如1000M，2G）（默认：1G）内存


• SPARK_WORKER_MEMORY 后，设置总内存的工人有多少给执行者（例如1000米2G）

• SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)
• SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
• SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)

什么是以上3个参数的关系？

what is the relationship of above 3 parameters?

据我了解，DRIVER_MEMORY是最大内存主节点/过程可以请求。但对于司机，怎么样多机的情况，例如。 1老爷机和2工人机器，工人该机也应该有一定的可用内存火花驱动程序？

As I understand, DRIVER_MEMORY is the max memory master node/process can request. But for driver, how about multiple machine situation, eg. 1 master machine and 2 worker machine, worker machine should also have some memory available for spark driver?

EXECUTOR_MEMORY和WORKER_MEMORY都是一样的我，只是叫法不同，可能这也解释好吗？

EXECUTOR_MEMORY and WORKER_MEMORY are the same to me, just different names, could this also be explained please?

非常感谢你。

推荐答案

首先，你应该知道，1 工人（你可以说1机或1个工作节点）可以启动多个执行人（或多个工人实例 - 他们在文档中使用的术语）

First, you should know that 1 Worker (you can say 1 machine or 1 Worker Node) can launch multiple Executors (or multiple Worker Instances - the term they use in the docs).



• SPARK_WORKER_MEMORY 仅在使用独立部署模式


• 在使用 SPARK_EXECUTOR_MEMORY YARN 部署模式


• SPARK_WORKER_MEMORY is only used in standalone deploy mode
• SPARK_EXECUTOR_MEMORY is used in YARN deploy mode

在单机模式下，您可以设置 SPARK_WORKER_MEMORY 的内存可使用的总金额一体机（所有本机上执行人），以运行你的应用程序的火花。

In Standalone mode, you set SPARK_WORKER_MEMORY to the total amount of memory can be used on one machine (All Executors on this machine) to run your spark applications.

在此相反，纱模式下，您可以设置 SPARK_DRIVER_MEMORY 来的记忆一名遗嘱执行人

In contrast, In YARN mode, you set SPARK_DRIVER_MEMORY to the memory of one Executor



• SPARK_DRIVER_MEMORY 纱线部署模式下使用，指定为运行你的应用程序和功放驱动程序内存;与群集管理器进行通信。


• SPARK_DRIVER_MEMORY is used in YARN deploy mode, specifying the memory for the Driver that runs your application & communicates with Cluster Manager.

这篇关于星火配置，是SPARK_DRIVER_MEMORY，SPARK_EXECUTOR_MEMORY和SPARK_WORKER_MEMORY的区别？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！