使用spark-submit，--total-executor-cores选项的行为是什么? [英] Using spark-submit, what is the behavior of the --total-executor-cores option?

问题描述

我正在通过python包装的C ++代码运行一个火花集群. 我目前正在测试多线程选项的不同配置(在Python级别或Spark级别).

我在HDFS 2.5.4群集上使用带有独立二进制文件的spark.该集群目前由10个从属组成，每个从属都有4个内核.

据我所见，默认情况下，Spark每个节点启动4个从属服务器(我一次有4个python在一个从属节点上工作).

如何限制此数字?我可以看到我为"spark-submit"提供了--total-executor-cores选项，但是很少有文档说明它如何影响集群中执行者的分布！

我将进行测试以弄清主意，但如果有学识的人知道此选项的作用，则可能会有所帮助.

更新:

我再次浏览了spark文档，这是我的理解:

• 默认情况下，我每个工作人员节点有一个执行者(这里有10个工作人员节点，因此有10个执行者)
• 但是，每个工作人员可以并行运行多个任务.在独立模式下，默认行为是使用所有可用的内核，这解释了为什么我可以观察到4个python.
• 要限制每个工作人员使用的内核数量并限制并行任务的数量，我至少有3个选择:
  • 使用--total-executor-cores和spark-submit(最不令人满意，因为没有关于如何处理核心池的线索)
  • 在配置文件中使用SPARK_WORKER_CORES
  • 在开始脚本中使用-c选项

本文档的以下几行 http://spark.apache.org/docs/latest/spark-standalone.html 帮助我弄清楚发生了什么事:

SPARK_WORKER_INSTANCES
每台计算机上要运行的工作程序实例数(默认值:1). 如果您拥有非常大的计算机并且想要多个Spark worker进程，则可以将其设置为大于1. 如果您确实设置了此设置，请确保还明确设置SPARK_WORKER_CORES以限制每个工作人员的核心，否则每个工作人员将尝试使用所有核心.

我仍然不清楚的是，在我的情况下，为什么最好将每个辅助节点的并行任务数限制为1，并依靠我的C ++旧代码多线程处理.完成学习后，我将用实验结果更新此帖子.

解决方案

文档似乎不清楚.

根据我的经验，最常见的分配资源的做法是指出执行者的数量和每个执行者的核心数量，例如(取自

I will run tests to get a clear idea, but if someone knowledgeable has a clue of what this option does, it could help.

Update :

I went through spark documentation again, here is what I understand :

• By default, I have one executor per worker node (here 10 workers node, hence 10 executors)
• However, each worker can run several tasks in parallel. In standalone mode, the default behavior is to use all available cores, which explains why I can observe 4 python.
• To limit the number of cores used per worker, and limit the number of parallel tasks, I have at least 3 options :
  • use --total-executor-cores whith spark-submit (least satisfactory, since there is no clue on how the pool of cores is dealt with)
  • use SPARK_WORKER_CORES in the configuration file
  • use -c options with the starting scripts

The following lines of this documentation http://spark.apache.org/docs/latest/spark-standalone.html helped me to figure out what is going on :

SPARK_WORKER_INSTANCES
Number of worker instances to run on each machine (default: 1). You can make this more than 1 if you have have very large machines and would like multiple Spark worker processes. If you do set this, make sure to also set SPARK_WORKER_CORES explicitly to limit the cores per worker, or else each worker will try to use all the cores.

What is still unclear to me is why it is better in my case to limit the number of parallel tasks per worker node to 1 and rely on my C++ legacy code multithreading. I will update this post with experiment results, when I will finish my study.

解决方案

The documentation does not seem clear.

From my experience, the most common practice to allocate resources is by indicating the number of executors and the number of cores per executor, for example (taken from here):

$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn-cluster \
--num-executors 10 \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 4 \
--queue thequeue \
lib/spark-examples*.jar \
10

However, this approach is limited to YARN, and is not applicable to standalone and mesos based Spark, according to this.

Instead, the parameter --total-executor-cores can be used, which represents the total amount of cores - of all executors - assigned to the Spark job. In your case, having a total of 40 cores, setting the attribute --total-executor-cores 40 would make use of all the available resources.

Unfortunately, I am not aware of how Spark distributes the workload when less resources than the total available are provided. If working with two or more simultaneous jobs, however, it should be transparent to the user, in that Spark (or whatever resource manager) would manage how the resources are managed depending on the user settings.

这篇关于使用spark-submit，--total-executor-cores选项的行为是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！