使用REST触发Spark作业 [英] Triggering spark jobs with REST
问题描述
我最近尝试了 apache spark .我的问题是更具体地触发火花作业. 这里我已经发布了有关了解火花工作的问题.在忙于工作之后,我继续着自己的要求.
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement.
我有一个REST端点,在这里我公开API来触发Jobs,我已经将Spring4.0用于Rest实施.现在,我想到了在Spring中实现Jobs as Service的工作,我将以编程方式提交Job,这意味着当端点触发时,使用给定的参数,我将触发该Job. 我现在有几个设计选项.
I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job. I have now few design options.
-
类似于下面的书面工作,我需要维护几个由抽象类调用的Jobs可能是
JobScheduler
.
/*Can this Code be abstracted from the application and written as
as a seperate job. Because my understanding is that the
Application code itself has to have the addJars embedded
which internally sparkContext takes care.*/
SparkConf sparkConf = new SparkConf().setAppName("MyApp").setJars(
new String[] { "/path/to/jar/submit/cluster" })
.setMaster("/url/of/master/node");
sparkConf.setSparkHome("/path/to/spark/");
sparkConf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.setLocalProperty("spark.scheduler.pool", "test");
// Application with Algorithm , transformations
延伸到上述位置的文件具有由服务处理的多个版本的作业.
extending above point have multiple versions of jobs handled by service.
或者使用Spark Job Server来执行此操作.
Or else use an Spark Job Server to do this.
首先,我想知道在这种情况下最佳的解决方案是明智的执行方式,同时还要明智地进行扩展.
Firstly, I would like to know what is the best solution in this case, execution wise and also scaling wise.
注意:我正在使用Spark的独立群集. 请帮忙.
Note : I am using a standalone cluster from spark. kindly help.
推荐答案
只需使用Spark JobServer https://github.com/spark-jobserver/spark-jobserver
Just use the Spark JobServer https://github.com/spark-jobserver/spark-jobserver
进行服务时要考虑很多事情,Spark JobServer已经涵盖了其中的大部分内容.如果您发现事情还不够好,那么发出请求并将代码添加到他们的系统中应该很容易,而不是从头开始重新发明它
There are a lot of things to consider with making a service, and the Spark JobServer has most of them covered already. If you find things that aren't good enough, it should be easy to make a request and add code to their system rather than reinventing it from scratch
这篇关于使用REST触发Spark作业的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!