如何从命令行设置Spark示例的主地址 [英] How to set Master address for Spark examples from command line

问题描述

注意： 作者正在寻找设置Spark Master的答案，以便在运行包含 no 更改源代码的Spark示例时设置Spark Master，而只选择选项如果可能的话，可以从命令行完成。

NOTE: They author is looking for answers to set the Spark Master when running Spark examples that involves no changes to the source code, but rather only options that can be done from the command-line if at all possible.

让我们考虑BinaryClassification示例的run（）方法：

Let us consider the run() method of the BinaryClassification example:

  def run(params: Params) {
    val conf = new SparkConf().setAppName(s"BinaryClassification with $params")
    val sc = new SparkContext(conf)

请注意，SparkConf没有提供任何手段配置SparkMaster。

Notice that the SparkConf did not provide any means to configure the SparkMaster.

当使用以下参数从Intellij运行该程序时：

When running this program from Intellij with the following arguments:

--algorithm LR --regType L2 --regParam 1.0 data/mllib/sample_binary_classification_data.txt

发生以下错误：

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set
in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:166)
    at org.apache.spark.examples.mllib.BinaryClassification$.run(BinaryClassification.scala:105)

我也试过加入反正Spark Master网址（虽然代码似乎不支持它..）

I have also tried adding in the Spark Master url anyways (though the code seems NOT to support it ..)

  spark://10.213.39.125:17088   --algorithm LR --regType L2 --regParam 1.0 
  data/mllib/sample_binary_classification_data.txt

和

--algorithm LR --regType L2 --regParam 1.0 spark://10.213.39.125:17088
data/mllib/sample_binary_classification_data.txt

两者都不适用于错误：

Error: Unknown argument 'data/mllib/sample_binary_classification_data.txt'

这里的参考是解析选项 - 它对Spa没有任何作用rkMaster：

For reference here is the options parsing - which does nothing with SparkMaster:

val parser = new OptionParser[Params]("BinaryClassification") {
  head("BinaryClassification: an example app for binary classification.")
  opt[Int]("numIterations")
    .text("number of iterations")
    .action((x, c) => c.copy(numIterations = x))
  opt[Double]("stepSize")
    .text(s"initial step size, default: ${defaultParams.stepSize}")
    .action((x, c) => c.copy(stepSize = x))
  opt[String]("algorithm")
    .text(s"algorithm (${Algorithm.values.mkString(",")}), " +
    s"default: ${defaultParams.algorithm}")
    .action((x, c) => c.copy(algorithm = Algorithm.withName(x)))
  opt[String]("regType")
    .text(s"regularization type (${RegType.values.mkString(",")}), " +
    s"default: ${defaultParams.regType}")
    .action((x, c) => c.copy(regType = RegType.withName(x)))
  opt[Double]("regParam")
    .text(s"regularization parameter, default: ${defaultParams.regParam}")
  arg[String]("<input>")
    .required()
    .text("input paths to labeled examples in LIBSVM format")
    .action((x, c) => c.copy(input = x))

所以..是..我可以继续修改源代码即但我怀疑我错过了一个可用的调整旋钮来完成这项工作，不涉及修改源代码。

So .. yes .. I could go ahead and modify the source code. But I suspect instead I am missing an available tuning knob to make this work that does not involve modifying the source code.

推荐答案

您可以通过添加JVM参数从命令行设置Spark master：

You can set the Spark master from the command-line by adding the JVM parameter:

-Dspark.master=spark://myhost:7077

这篇关于如何从命令行设置Spark示例的主地址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！