如何使用Apache Livy设置Spark配置属性? [英] How to set Spark configuration properties using Apache Livy?

问题描述

在将Spark作业提交给Apache Livy时，我不知道如何以编程方式传递SparkSession参数:

I don't know how to pass SparkSession parameters programmatically when submitting Spark job to Apache Livy:

这是Test Spark作业:

This is the Test Spark job:

class Test extends Job[Int]{

  override def call(jc: JobContext): Int = {

    val spark = jc.sparkSession()

    // ...

  }
}

这是将Spark作业提交给Livy的方式:

This is how this Spark job is submitted to Livy:

val client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .build()

try {
  client.uploadJar(new File(testJarPath)).get()

  client.submit(new Test())

} finally {
  client.stop(true)
}

如何将以下配置参数传递给SparkSession?

How can I pass the following configuration parameters to SparkSession?

  .config("es.nodes","1localhost")
  .config("es.port",9200)
  .config("es.nodes.wan.only","true")
  .config("es.index.auto.create","true")

推荐答案

您可以像这样通过LivyClientBuilder轻松地做到这一点:

You can do that easily through the LivyClientBuilder like this:

val client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .setConf("es.nodes","1localhost")
  .setConf("key", "value")
  .build()

这篇关于如何使用Apache Livy设置Spark配置属性?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！