在YARN上运行Spark应用程序，无需提交火花 [英] Running a Spark application on YARN, without spark-submit

问题描述

我知道可以使用 spark-submit --master yarn 在YARN上执行Spark应用程序.

I know that Spark applications can be executed on YARN using spark-submit --master yarn.

问题是:是否可以使用 yarn 命令在纱线上运行Spark应用程序?

The question is: is it possible to run a Spark application on yarn using the yarn command ?

如果是这样，可以将YARN REST API用作以统一方式运行spark和MapReduce应用程序的接口.

If so, the YARN REST API could be used as interface for running spark and MapReduce applications in a uniform way.

推荐答案

我看到这个问题已经一岁了，但是对于任何偶然发现这个问题的人来说，现在看来应该可以这样做了.我一直在尝试做类似的事情，并一直在尝试遵循

I see this question is a year old, but to anyone else who stumbles across this question it looks like this should be possible now. I've been trying to do something similar and have been attempting to follow the Starting Spark jobs directly via YARN REST API Tutorial from Hortonworks.

基本上，您需要做的是将jar上传到HDFS，根据YARN REST API文档创建一个Spark Job JSON文件，然后使用curl命令启动该应用程序.该命令的一个示例是:

Essentially what you need to do is upload your jar to HDFS, create a Spark Job JSON file per the YARN REST API Documentation, and then use a curl command to start the application. An example of that command is:

curl -s -i -X POST -H "Content-Type: application/json" ${HADOOP_RM}/ws/v1/cluster/apps \
     --data-binary spark-yarn.json 

这篇关于在YARN上运行Spark应用程序，无需提交火花的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！