纱线概念理解的火花 [英] Spark on yarn concept understanding

问题描述

我想了解 spark 如何在 YARN 集群/客户端上运行.我有以下疑问.

I am trying to understand how spark runs on YARN cluster/client. I have the following question in my mind.

1. 是否需要在yarn集群的所有节点上都安装spark?我认为应该是因为集群中的工作节点执行任务并且应该能够解码驱动程序发送到集群的 spark 应用程序中的代码(spark API)?

1. Is it necessary that spark is installed on all the nodes in yarn cluster? I think it should because worker nodes in cluster execute a task and should be able to decode the code(spark APIs) in spark application sent to cluster by the driver?

它在文档中说确保 HADOOP_CONF_DIR 或 YARN_CONF_DIR 指向包含 Hadoop 集群(客户端)配置文件的目录".为什么客户端节点在向集群发送作业时必须安装Hadoop?

It says in the documentation "Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster". Why does client node have to install Hadoop when it is sending the job to cluster?

推荐答案

我们正在 YARN 上运行 Spark 作业(我们使用 HDP 2.2).

We are running spark jobs on YARN (we use HDP 2.2).

我们没有在集群上安装 spark.我们只将 Spark 程序集 jar 添加到 HDFS.

We don't have spark installed on the cluster. We only added the Spark assembly jar to the HDFS.

例如运行 Pi 示例:

For example to run the Pi example:

./bin/spark-submit \
  --verbose \
  --class org.apache.spark.examples.SparkPi \
  --master yarn-cluster \
  --conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar \
  --num-executors 2 \
  --driver-memory 512m \
  --executor-memory 512m \
  --executor-cores 4 \
  hdfs://master:8020/spark/spark-examples-1.3.1-hadoop2.6.0.jar 100

--conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar - 这个配置告诉纱线来自拿火花总成.如果你不使用它，它会上传你运行 spark-submit 的 jar.

--conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar - This config tell the yarn from were to take the spark assembly. If you don't use it, it will upload the jar from were you run spark-submit.

关于您的第二个问题:客户端节点不需要安装 Hadoop.它只需要配置文件.您可以将目录从集群复制到客户端.

About your second question: The client node doesn't not need Hadoop installed. It only needs the configuration files. You can copy the directory from your cluster to your client.

这篇关于纱线概念理解的火花的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！