如何将Livy curl调用转换为Livy Rest API调用 [英] How to convert Livy curl call to Livy Rest API call

问题描述

我开始使用Livy，在我的设置中，Livy服务器在Unix机器上运行，我可以对它进行卷曲并执行作业.我创建了一个胖子罐并将其上传到hdfs，我只是从Livy调用了它的主要方法.我的用于Livy的Json有效负载如下所示:

I am getting started with Livy, in my setup Livy server is running on Unix machine and I am able to do curl to it and execute the job. I have created a fat jar and uploaded it on hdfs and I am simply calling its main method from Livy. My Json payload for Livy looks like below:

{
"file" : "hdfs:///user/data/restcheck/spark_job_2.11-3.0.0-RC1-
SNAPSHOT.jar",
"proxyUser" : "test_user",
"className" : "com.local.test.spark.pipeline.path.LivyTest",
"files" : ["hdfs:///user/data/restcheck/hivesite.xml","hdfs:///user/data/restcheck/log4j.properties"],
"driverMemory" : "5G",
"executorMemory" : "10G",
"executorCores" : 5,
"numExecutors" : 10,
"queue" : "user.queue",
"name"  : "LivySampleTest2",
"conf" : {"spark.master" : "yarn","spark.executor.extraClassPath" : 
"/etc/hbase/conf/","spark.executor.extraJavaOptions" : "-Dlog4j.configuration=file:log4j.properties","spark.driver.extraJavaOptions" : "-Dlog4j.configuration=file:log4j.properties","spark.ui.port" : 4100,"spark.port.maxRetries" : 100,"JAVA_HOME" : "/usr/java/jdk1.8.0_60","HADOOP_CONF_DIR" : 
"/etc/hadoop/conf:/etc/hive/conf:/etc/hbase/conf","HIVE_CONF_DIR" : 
"/etc/hive/conf"}
}

和下面是我对它的呼唤:

and below is my curl call to it:

curl -X POST --negotiate -u:"test_user" --data @/user/data/Livy/SampleFile.json  -H "Content-Type: application/json" https://livyhost:8998/batches 

我正在尝试将其转换为REST API调用，并遵循

I am trying to convert this a REST API call and following the WordCount example provided by Cloudera but not able to covert my curl call to the REST API. I have all the jars already added in HDFS so I dont think I need to do the upload jar call.

推荐答案

它也应该与curl一起使用

It should work with curl also

请尝试以下JSON.

curl -H "Content-Type: application/json" https://livyhost:8998/batches
 -X POST --data '{
  "name" : "LivyREST",
  "className" :  "com.local.test.spark.pipeline.path.LivyTest",
  "file"  : "/user/data/restcheck/spark_job_2.11-3.0.0-RC1-
SNAPSHOT.jar"
}' 

我还要添加更多参考

http://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-submitting-batch-jar-python- and-streaming-spark-jobs/

这篇关于如何将Livy curl调用转换为Livy Rest API调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！