无法在火花中的commmandline上覆盖Typesafe配置 [英] cant override Typesafe configuration on commmandline in spark

问题描述

我在src/main/resources文件夹中有一个类型安全配置application.conf，默认情况下已加载.

I have a typesafe configuration application.conf in the src/main/resourcesfolder which is loaded by default.

可以通过指定以下值来覆盖单个值:

A single value can be overridden by specifying:

--conf spark.driver.extraJavaOptions=-DsomeValue="foo"

但是，请指定一个全新的文件，即覆盖application.conf文件，例如:

However, specifying a complete new, i.e. overriding application.conf file like:

spark-submit \
    --class my.Class \
    --master "local[2]" \
    --files foo.conf \
    --conf spark.driver.extraClassPath="-Dconfig.file=file:foo.conf" \
    --conf spark.driver.extraJavaOptions=-Dvalue="abcd" \
    job.jar

将无法加载foo.conf.而是从资源文件夹中加载原始文件. 尝试使用以下技巧:在纱线上使用Spark时使用类型安全配置没有也有帮助.

will fail to load foo.conf. Instead, the original file from the resources folder will be loaded. Trying the tricks from: Using typesafe config with Spark on Yarn did not help as well.

覆盖多个配置使用uberjar进行部署时，Typesafe配置中的值似乎是普通(无火花)程序的答案. 问题仍然是如何激发这一点.

Overriding multiple config values in Typesafe config when using an uberjar to deploy seems to be the answer for plain (without spark) programs. The question remains how to bring this to spark.

也通过:

--conf spark.driver.extraClassPath="-Dconfig.resource=file:foo.conf"
--conf spark.driver.extraClassPath="-Dconfig.resource=foo.conf"

无法从命令行加载我的配置.

fails to load my configuration from the command line .

不过，根据文档:

https://github.com/lightbend/config 适用于使用 应用程序.{conf，json，properties}，系统属性可用于 强制使用其他配置源(例如，从命令行 -Dconfig.file =路径/到/配置文件):

https://github.com/lightbend/config For applications using application.{conf,json,properties}, system properties can be used to force a different config source (e.g. from command line -Dconfig.file=path/to/config-file):

• config.resource指定资源名称-不是基本名称，即application.conf不是应用程序
• config.file指定了文件系统路径，再次应包含扩展名，而不是基本名称
• config.url指定一个URL

• config.resource specifies a resource name - not a basename, i.e. application.conf not application
• config.file specifies a filesystem path, again it should include the extension, not be a basename
• config.url specifies a URL

这些系统属性指定替换 应用程序.{conf，json，properties}，而不是附加项.他们只会影响 使用默认ConfigFactory.load()配置的应用程序.在里面 替换配置文件，您可以使用include"application"来包含 原始的默认配置文件；在include语句之后，您 可以继续覆盖某些设置.

These system properties specify a replacement for application.{conf,json,properties}, not an addition. They only affect apps using the default ConfigFactory.load() configuration. In the replacement config file, you can use include "application" to include the original default config file; after the include statement you could go on to override certain settings.

使用这些参数应该是可能的.

it should be possible with these parameters.

推荐答案

spark-submit \
    --class my.Class \
    --master "local[2]" \
    --files foo.conf \
    --conf spark.driver.extraJavaOptions="-Dvalue='abcd' -Dconfig.file=foo.conf" \
    target/scala-2.11/jar-0.1-SNAPSHOT.jar

从spark.driver.extraClassPath更改为spark.driver.extraJavaOptions可以解决问题

这篇关于无法在火花中的commmandline上覆盖Typesafe配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！