在Eclipse中运行火花code火花正在安装另一台服务器上 [英] Running spark code on eclipse with spark being install on another server

问题描述

我已经配置了斯卡拉Eclipse和创建一个Maven项目，而在Windows写了一个简单的字数火花的工作。现在我的火花+ Hadoop的安装Linux服务器。如何从Eclipse启动我的火花code火花簇（这是在Linux上）？

I have configured eclipse for scala and created a maven project and wrote a simple word count spark job on windows. Now my spark+hadoop are installed on linux server. How can I launch my spark code from eclipse to spark cluster (which is on linux)?

任何建议。

推荐答案

其实这个答案并非如此简单，你所期望的。

Actually this answer is not so simple, as you would expect.

我在许多假设，首先你使用 SBT ，第二个是，你是在一个基于Linux的计算机工作，第三是最后被你有两个类在您的项目，比方说邵仁枚和全局，和最后的假设是你想要设置里面的程序设置。因此，在某个地方你可运行code你必须有这样的事情：

I will make many assumptions, first that you use sbt, second is that you are working in a linux based computer, third is the last is that you have two classes in your project, let's say RunMe and Globals, and the last assumption will be that you want to set up the settings inside the program. Thus, somewhere in your runnable code you must have something like this:

object RunMe {
  def main(args: Array[String]) {
    val conf = new SparkConf()
      .setMaster("mesos://master:5050") //If you use Mesos, and if your network resolves the hostname master to its IP.
      .setAppName("my-app")
      .set("spark.executor.memory", "10g")
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext()

    //your code comes here
  }
}

您必须遵循的步骤如下：

The steps you must follow are:

• 编译工程，在它的根，通过使用：

• Compile the project, in the root of it, by using:

$ SBT组装

将作业发送到主节点，这是有趣的部分（假设你有一个结构项目中的目标/斯卡拉/ ，和里面你有一个文件的.jar ，对应于编译的项目）

Send the job to the master node, this is the interesting part (assuming you have the next structure in your project target/scala/, and inside you have a file .jar, which corresponds to the compiled project)

$火花提交--class邵仁枚的目标/斯卡拉/ app.jar

注意，因为我认为该项目有你必须确定你想要运行的类两个或多个类。此外，我敢打赌，这两种方法，为纱线和 Mesos 非常相似。

Notice that, because I assumed that the project has two or more classes you would have to identify which class you want to run. Furthermore, I bet that both approaches, for Yarn and Mesos are very similar.

这篇关于在Eclipse中运行火花code火花正在安装另一台服务器上的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！