具有多个执行程序的Spark独立配置 [英] Spark standalone configuration having multiple executors

问题描述

我正在尝试设置独立的Spark 2.0服务器以并行处理分析功能.为此，我想让一个工人有多个执行者.

I'm trying to setup a standalone Spark 2.0 server to process an analytics function in parallel. To do this I want to have a single worker with multiple executors.

我正在使用:

• 独立Spark 2.0
• 8核
• 24gig RAM
• Windows Server 2008
• pyspark(尽管这似乎无关)

这仅仅是出于概念证明的目的，但是我希望有8个执行器，每个核心一个.

This is just for pure proof of concept purposes but I want to have 8 executors, one per each core.

我尝试关注该主题的其他主题，但由于某些原因，它不适用于我. IE: Spark独立编号执行器/Cores控制

I've tried to follow the other threads on this topic but for some reason it's not working for me. IE: Spark Standalone Number Executors/Cores Control

我的配置如下:

conf \ spark-defaults.conf

spark.cores.max = 8
spark.executor.cores = 1

我也试图将我的spark-env.sh文件更改为无效.相反，发生的事情是它表明我的1个工人上只有1个执行者.如下所示，它仍然显示具有1个执行器和8个内核的独立执行器.

I have tried to also change my spark-env.sh file to no avail. Instead what is happening is that it shows that my 1 worker only has 1 executor on it. As you can see below, it still shows the standalone with 1 executor with 8 cores to it.

推荐答案

我相信您混合了本地模式和独立模式:

I believe you mixed up local and standalone modes:

• 本地模式是一种开发工具，其中所有进程都在单个JVM中执行.通过将master设置为local，local[*]或local[n]，可以在本地模式下启动应用程序. spark.executor.cores和spark.executor.cores在本地模式下不适用，因为只有一个嵌入式执行程序.

• 独立模式需要独立的 Spark集群.它需要一个主节点(可以使用SPARK_HOME/sbin/start-master.sh脚本启动)和至少一个工作节点(可以使用SPARK_HOME/sbin/start-slave.sh脚本启动).

• Local mode is a development tool where all processes are executed inside a single JVM. Application is started in a local mode by setting master to local, local[*] or local[n]. spark.executor.cores and spark.executor.cores are not applicable in the local mode because there is only one embedded executor.

• Standalone mode requires a standalone Spark cluster. It requires a master node (can be started using SPARK_HOME/sbin/start-master.sh script) and at least one worker node (can be started using SPARK_HOME/sbin/start-slave.sh script).

SparkConf应该使用主节点地址来创建(spark://host:port).

SparkConf should use master node address to create (spark://host:port).

这篇关于具有多个执行程序的Spark独立配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！