如何使用Spark-Submit在集群模式下将环境变量传递给Spark驱动程序 [英] How to pass environment variables to spark driver in cluster mode with spark-submit

问题描述

spark-submit允许使用--conf spark.executorEnv.FOO=bar配置执行程序环境变量，Spark REST API允许使用environmentVariables字段传递某些环境变量. 不幸的是，在集群模式下使用spark-submit提交驱动程序时，我发现没有任何与配置驱动程序的环境变量相似的事情:

spark-submit allows to configure the executor environment variables with --conf spark.executorEnv.FOO=bar, and the Spark REST API allows to pass some environment variables with the environmentVariables field. Unfortunately I've found nothing similar to configure the environment variable of the driver when submitting the driver with spark-submit in cluster mode:

spark-submit --deploy-mode cluster myapp.jar

是否可以在群集模式下使用spark-submit设置驱动程序的环境变量?

Is it possible to set the environment variables of the driver with spark-submit in cluster mode?

推荐答案

至少在YARN上有效，

On YARN at least, this works:

spark-submit --deploy-mode cluster --conf spark.yarn.appMasterEnv.FOO=bar myapp.jar

http://spark.apache.org/docs中已提及/latest/configuration.html#environment-variables :

注意:在cluster模式下在YARN上运行Spark时，需要使用conf/spark-defaults.conf文件中的spark.yarn.appMasterEnv.[EnvironmentVariableName]属性设置环境变量.

Note: When running Spark on YARN in cluster mode, environment variables need to be set using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file.

我已经测试过它可以与spark-submit的--conf标志一起传递，这样您就不必编辑全局conf文件.

I have tested that it can be passed with --conf flag for spark-submit, so that you don't have to edit global conf files.

这篇关于如何使用Spark-Submit在集群模式下将环境变量传递给Spark驱动程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！