加载Spark Config以测试Spark应用程序 [英] Loading Spark Config for testing Spark Applications

问题描述

我一直在尝试在本地笔记本电脑上测试Spark应用程序，然后再将其部署到群集上(以避免每次都打包和部署我的整个应用程序)，但是在加载spark配置文件时却很费力.

I've been trying to test a spark application on my local laptop before deploying it to a cluster (to avoid having to package and deploy my entire application every time) but struggling on loading the spark config file.

当我在集群上运行应用程序时，通常会向应用程序提供一个火花配置文件(使用spark-submit的--conf).该文件具有许多配置选项，因为此应用程序与Cassandra和HDFS进行交互.但是，当我尝试在本地笔记本电脑上执行相同操作时，我不确定如何加载此配置文件.我知道我可能可以编写一段代码来获取配置文件的文件路径，然后遍历并解析所有值并在配置中进行设置，但是我只是想知道是否有更简单的方法.

When I run my application on a cluster, I am usually providing a spark config file to the application (using spark-submit's --conf). This file has a lot of config options because this application interacts with Cassandra and HDFS. However, when I try to do the same on my local laptop, I'm not sure exactly how to load this config file. I know I can probably write a piece of code that takes the file path of the config file and just goes through and parses all the values and sets them in the config, but I'm just wondering if there are easier ways.

当前状态:

• 我将所需的配置文件放在我的SPARK_HOME/conf目录中，并将其命名为spark-defaults.conf --->并没有应用，但是使用spark-submit可以运行相同的文件
• 对于本地模式，当我创建spark会话时，我将Spark Master设置为"local [2]".我在创建spark会话时正在执行此操作，所以我想知道是否可以使用指定的配置文件创建此会话.

推荐答案

不确定是否对任何人有帮助，但是我最终从测试资源目录中读取conf文件，然后将所有值设置为系统属性(复制了此文件).来自Spark源代码):

Not sure if this will help anyone, but I ended up reading the conf file from a test resource directory and then setting all the values as system properties (copied this from Spark Source Code):

//_sparkConfs is just a map of (String,String) populated from reading the conf file
for {
  (k, v) ← _sparkConfs
} {
  System.setProperty(k, v)
}

这实际上是在一定程度上模拟spark-submit的--properties-file选项.这样，我可以在测试设置中保留此逻辑，而无需修改现有的应用程序代码.

This is essentially emulating the --properties-file option of spark-submit to a certain degree. By doing this, I was able to keep this logic in my test setup, and not need to modify the existing application code.

这篇关于加载Spark Config以测试Spark应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！