在纱概念理解上的星火 [英] Spark on yarn concept understanding

问题描述

我试图理解在YARN集群/客户端上如何运行spark。在我看来，我有以下问题。

1. 是否有必要在纱线群中的所有节点上安装spark？我认为它应该是因为集群中的工作节点执行任务，并且应该能够解码驱动程序发送到集群的spark应用程序中的代码（spark API）？

p>它在文档确保 HADOOP_CONF_DIR 或 YARN_CONF_DIR 指向包含（客户端） Hadoop集群的配置文件。为什么客户机节点在将作业发送到群集时必须安装Hadoop？

我们正在YARN上运行spark任务（我们使用HDP 2.2）。

我们没有在集群上安装spark。我们只将Spark组件jar添加到HDFS中。

例如，运行Pi示例：

<$ p $
--class org.apache.spark.examples.SparkPi \
- master yarn-cluster \
--conf spark.yarn.jar = hdfs：// master：8020 / spark / spark-assembly-1.3.1-hadoop2.6.0.jar \
- num-executors 2 \
--driver-memory 512m \
--executor -memory 512m \
--executor-cores 4 \
hdfs：// master：8020 / spark / spark-examples-1.3.1-hadoop2.6.0.jar 100

- conf spark.yarn.jar = hdfs：// master：8020 / spark / spark-assembly-1.3.1-hadoop2.6.0.jar - 此配置告诉纱线从中取出火花组件。如果你不使用它，它会从你运行 spark-submit 的时候上传jar。

关于你的第二个问题：客户端节点不需要安装Hadoop。它只需要配置文件。您可以将目录从群集复制到客户端。

I am trying to understand how spark runs on YARN cluster/client. I have the following question in my mind.

1. Is it necessary that spark is installed on all the nodes in yarn cluster? I think it should because worker nodes in cluster execute a task and should be able to decode the code(spark APIs) in spark application sent to cluster by the driver?

2. It says in the documentation "Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster". Why does client node have to install Hadoop when it is sending the job to cluster?

解决方案

We are running spark jobs on YARN (we use HDP 2.2).

We don't have spark installed on the cluster. We only added the Spark assembly jar to the HDFS.

For example to run the Pi example:

./bin/spark-submit \
  --verbose \
  --class org.apache.spark.examples.SparkPi \
  --master yarn-cluster \
  --conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar \
  --num-executors 2 \
  --driver-memory 512m \
  --executor-memory 512m \
  --executor-cores 4 \
  hdfs://master:8020/spark/spark-examples-1.3.1-hadoop2.6.0.jar 100

--conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar - This config tell the yarn from were to take the spark assembly. If you don't use it, it will upload the jar from were you run spark-submit.

About your second question: The client node doesn't not need Hadoop installed. It only needs the configuration files. You can copy the directory from your cluster to your client.

这篇关于在纱概念理解上的星火的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！