如何使它更易于部署我的罐子星火群集在独立模式？ [英] How to make it easier to deploy my Jar to Spark Cluster in standalone mode?

问题描述

我对开发和测试的小集群3台机器，而另一台机器。开发时，我设置 SparkContext 到本地。当一切OK，我要部署的Jar文件我建到每一个节点。基本上我手动移动这个jar集群，并复制到集群公用HDFS。然后，我可以在code更改为：

I have a small cluster with 3 machines, and another machine for developing and testing. When developing, I set SparkContext to local. When everything is OK, I want to deploy the Jar file I build to every node. Basically I manually move this jar to cluster and copy to HDFS which shared by the cluster. Then I could change the code to:

//standalone mode
val sc = new SparkContext(
     "spark://mymaster:7077", 
     "Simple App", 
     "/opt/spark-0.9.1-bin-cdh4",   //spark home
     List("hdfs://namenode:8020/runnableJars/SimplyApp.jar") //jar location
) 

在我的IDE中运行它。我的问题：有没有办法更容易移动这个罐子集群

to run it in my IDE. My question: Is there any way easier to move this jar to cluster?

推荐答案

在星火，该程序创建SparkContext被称为司机。这是足够的，与你的工作jar文件提供给驾驶者的本地文件系统，以便它把它捡起来并运到主/工人。

In Spark, the program creating the SparkContext is called 'the driver'. It's sufficient that the jar file with your job is available to the local file system of the driver in order for it to pick it up and ship it to the master/workers.

在具体的，你的配置看起来像：

In concrete, your config will look like:

//favor using Spark Conf to configure your Spark Context
val conf = new SparkConf()
             .setMaster("spark://mymaster:7077")
             .setAppName("SimpleApp")
             .set("spark.local.ip", "172.17.0.1")
             .setJars(Array("/local/dir/SimplyApp.jar"))

val sc = new SparkContext(conf)

引擎盖下，驱动程序将启动一个服务器里的工人将下载从驱动程序的jar文件（S）。这是非常重要的（通常一个问题），就是工人已经给司机的网络访问。这通常可以通过设置spark.local.ip在网络上这是访问/路由从工人在驱动程序来保证。

Under the hood, the driver will start a server where the workers will download the jar file(s) from the driver. It's therefore important (and often an issue) that the workers have network access to the driver. This can often be ensured by setting 'spark.local.ip' on the driver in a network that's accessible/routable from the workers.

这篇关于如何使它更易于部署我的罐子星火群集在独立模式？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！