如何通过REST API提交工作? [英] How to submit a job via REST API?

问题描述

我正在使用Datastax Enterprise 4.8.3. 我正在尝试实现基于Quartz的应用程序以远程提交Spark作业. 在研究期间，我偶然发现了以下链接:

I'm using Datastax Enterprise 4.8.3. I'm trying to implement a Quartz based application to remotely submit Spark jobs. During my research I have stumbled upon the following links:

1. Apache Spark隐藏的REST API
2. Spark功能-在独立群集模式下提供稳定的应用程序提交网关

1. Apache Spark Hidden REST API
2. Spark feature - Provide a stable application submission gateway in standalone cluster mode

为了检验理论，我尝试在我的2节点集群(如上面的链接#1中提供)的主节点(IP:"spark-master-ip"；直接在外壳上)上执行以下代码片段:

To test out the theory I tried executing the below code snippet on the master node (IP: "spark-master-ip"; directly on the shell) of my 2 node cluster (as provided in link #1 above):

curl -X POST http://spark-master-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
    "action" : "CreateSubmissionRequest",
    "appArgs" : [ "myAppArgument1" ],
    "appResource" : "file:/home/local/sparkjob.jar",
    "clientSparkVersion" : "1.4.2",
    "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "com.spark.job.Launcher",
  "sparkProperties" : {
      "spark.jars" : "file:/home/local/sparkjob.jar",
      "spark.driver.supervise" : "false",
      "spark.app.name" : "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.submit.deployMode" : "cluster",
      "spark.master" : "spark://spark-master-ip:6066"
  }
}'

但是执行代码后，我得到了一个带有以下文本的html响应:

But executing the code I get an html response with the following text:

This Page Cannot Be Displayed
The system cannot communicate with the external server (spark-master-ip).
The Internet server may be busy, may be permanently down, or may be unreachable because of network problems.
Please check the spelling of the Internet address entered.
If it is correct, try this request later.

If you have questions, please contact your organization's network administrator and provide the codes shown below.

Date: Fri, 11 Dec 2015 13:19:15 GMT
Username: 
Source IP: spark-master-ip
URL: POST http://spark-master-ip/v1/submissions/create
Category: Uncategorized URLs
Reason: UNKNOWN
Notification: GATEWAY_TIMEOUT

推荐答案

• 检查您是否已启动Spark master和worker(使用start-all.sh)

  检查日志文件中是否有类似

  Check that in the log file there is a message like

  INFO rest.StandaloneRestServer:已启动用于在端口6066上提交应用程序的REST服务器

  INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066

  • 检查已启动的进程是否真的在端口6066上侦听(使用netstat)
  • Check the started process is really listening on port 6066 (using netstat)

  它应该看起来像这样:

  
  root@x:~# netstat -apn | grep 11572 | grep LISTEN
  tcp6       0      0 :::8080                 :::*                    LISTEN      11572/java      
  tcp6       0      0 10.0.0.9:6066           :::*                    LISTEN      11572/java      
  tcp6       0      0 10.0.0.9:7077           :::*                    LISTEN      11572/java      
  

  然后将脚本中的"spark-master-ip"替换为您在netstat输出中看到的IP地址(示例显示为"10.0.0.9").

  Then replace "spark-master-ip" in the script with the IP address you see in the output of netstat (the example shows "10.0.0.9").

  这篇关于如何通过REST API提交工作?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！