Spark - 使用参数从java调用Spark jar [英] Spark - Call Spark jar from java with arguments

问题描述

我想从java调用spark jar（在yarn上运行spark进程），并尝试使用链接代码。

I would like to call spark jar from java (to run spark process on yarn), and try to use this link code.

它看起来很适合我的情况，但我需要将hashmap和一些java值传递给spakr jar。它是否能够将java对象传递给spark jar？

It looks fit in my case, but I need to pass hashmap and some java values to spakr jar. Is it able to pass java object to spark jar?

并且java方面能够知道如何使用spark jar进程或者它是如何完成的？如果是这样，怎么样？

And is java side able to know how mush spark jar process or is it done? if so, how?

推荐答案

我认为你误解了数据算法中给出的内容

I think you misunderstood the content given in data-algorithms

有两种提交作业的方式

1）从shell脚本中提交类似Spark的提交

1) Spark-submit like below example from shell script

cat run_secondarysorting.sh

#!/bin/bash
export JAVA_HOME=/usr/java/jdk7
export SPARK_HOME=/home/hadoop/spark-1.1.0
export SPARK_MASTER=spark://myserver100:7077
BOOK_HOME=/home/mp/data-algorithms-book
APP_JAR=$BOOK_HOME/dist/data_algorithms_book.jar
INPUT=/home/hadoop/testspark/timeseries.txt
# Run on a Spark standalone cluster
prog=org.dataalgorithms.chap01.spark.SparkSecondarySort
$SPARK_HOME/bin/spark-submit \
--class $prog \
--master $SPARK_MASTER \
--executor-memory 2G \
--total-executor-cores 20 \
$APP_JAR

2）来自Yarn Client在链接中描述。

2) From Yarn Client which was described in the link.

Yarn客户端的用法是

Usage of Yarn Client is

想要从Java代码提交Spark作业（例如Java servlets或
其他Java代码）例如REST服务器。）

want to submit Spark jobs from Java code (such as Java servlets or other Java code such as REST servers).

当你打电话给这个纱线客户端...那么你需要打电话作为休息服务或servlet等方法...（即通过web），你也可以在其中传递HashMap或任何java对象类型的参数......

When you are calling this yarn client... then you need to call as method in your rest-service or servlet etc... (i.e. through web) in which you can also pass parameters like HashMap or any java object kind...

出于演示目的已编写独立客户端（公共静态无效主体）

希望您理解..

这篇关于Spark - 使用参数从java调用Spark jar的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！