创建独立的按需 Apache Spark Web 服务 [英] Creating a standalone on-demand Apache Spark web service

问题描述

我对网络服务完全陌生，我正在尝试创建一个像这样的网络服务:

I am completely new to web services and I am trying to create a web service which goes like this:

用户在网页中输入一些字词.在后端启动一个 Apache Spark 作业，它使用这些词作为输入参数这项工作的结果将被处理并显示在网页上.我想让它成为一个独立的东西，而不使用任何 Spark 即服务产品，如 Bluemix 等

The user enters some words into a webpage. An Apache Spark job is started in the backend which uses these words as input parameters The results of this job are to be processed and displayed back in the webpage. I want to make this a standalone thing without using any Spark-as-a-service offerings like Bluemix, etc

现在，我正在考虑采用这样的方法:后端的 Python 服务器接受请求并使用输入启动新的 Spark 作业.当 spark 作业完成时，Python 服务会处理输出并将其发送回页面.

Right now, I am thinking of going with something like this: a Python server in the backend accepts requests and fires off a new Spark job with the input. When the spark job completes, the Python service processes the output and sends it back to the page.

我不确定这是否是最好的方法.另外，我不确定如何在网页中实时显示 Spark 作业的进度.

I am not sure if this is the best way to go about doing this. Also, I'm not sure how to display the progress of the Spark job in real time in the web page.

推荐答案

apache livy 项目可能适合您的需求.

The apache livy project might fit your needs.

https://livy.incubator.apache.org/

Livy 支持通过 REST 接口或 RPC 客户端库使用同步和异步结果提交 Spark.

Livy enables submission of Spark using synchronous and asynchronous result via a REST interface or a RPC client library.

希望对您有所帮助！

这篇关于创建独立的按需 Apache Spark Web 服务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！