如何使用 spark-submit 在集群模式下将环境变量传递给 spark 驱动程序 [英] How to pass environment variables to spark driver in cluster mode with spark-submit

问题描述

spark-submit 允许使用 --conf spark.executorEnv.FOO=bar 配置执行器环境变量，Spark REST API 允许传递一些环境变量environmentVariables 字段.不幸的是，在集群模式下使用 spark-submit 提交驱动程序时，我没有发现与配置驱动程序环境变量类似的东西:

spark-submit allows to configure the executor environment variables with --conf spark.executorEnv.FOO=bar, and the Spark REST API allows to pass some environment variables with the environmentVariables field. Unfortunately I've found nothing similar to configure the environment variable of the driver when submitting the driver with spark-submit in cluster mode:

spark-submit --deploy-mode cluster myapp.jar

是否可以在集群模式下用spark-submit设置驱动的环境变量?

Is it possible to set the environment variables of the driver with spark-submit in cluster mode?

推荐答案

至少在 YARN 上，这是可行的:

On YARN at least, this works:

spark-submit --deploy-mode cluster --conf spark.yarn.appMasterEnv.FOO=bar myapp.jar

在 http://spark.apache.org/docs/latest/configuration.html#environment-variables 那:

注意:在 cluster 模式下在 YARN 上运行 Spark 时，需要使用 spark.yarn.appMasterEnv.[EnvironmentVariableName] 属性设置环境变量>conf/spark-defaults.conf 文件.

Note: When running Spark on YARN in cluster mode, environment variables need to be set using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file.

我已经测试过它可以通过 spark-submit 的 --conf 标志传递，这样你就不必编辑全局 conf 文件了.

I have tested that it can be passed with --conf flag for spark-submit, so that you don't have to edit global conf files.

这篇关于如何使用 spark-submit 在集群模式下将环境变量传递给 spark 驱动程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！