应用程序配置文件中的setMaster与sparkSubmit上的--master标志之间存在火花差异或冲突 [英] Spark difference or conflicts between setMaster in app conf and --master flag on sparkSubmit

问题描述

我试图理解在运行Spark应用程序时设置master属性的重要性.

I'm trying to understand the importance of setting the master property when running a spark application.

群集位置在默认端口7077上.我正在测试机上运行此应用程序，它将在其中撞到s3存储桶.

The cluster location is at the default port of 7077. I'm running this app from a testmachine where it will hit an s3 bucket.

当前应用程序中的spark配置显示为:

Currently spark configuration in the app reads:

val sparkConf = new SparkConf()
.setMaster("spark://127.0.0.1:7077")  

但是我还在命令行上使用spark Submit设置了标志:

but I'm also setting the flag on the command line with spark submit:

--master spark://127.0.0.1:7077

那么，同时设置这两个设置是否会引起问题?一个会被另一个覆盖吗?他们都是必要的吗?

So, does having both of these set cause problems? Does one get overridden by the other? Are they both necessary?

推荐答案

那么，同时设置这两个设置是否会引起问题?一个得到 被其他覆盖?他们都是必要的吗?

So, does having both of these set cause problems? Does one get overridden by the other? Are they both necessary?

火花配置" 页非常清晰(强调我的意思):

The Spark Configuration page is very clear (emphasis mine):

将传递任何指定为标志或属性文件中的值 进入应用程序，并与通过以下方式指定的应用程序合并 SparkConf. 直接在SparkConf上设置的属性占最高 优先级，然后将标志传递到spark-submit或spark-shell，然后 spark-defaults.conf文件中的所有选项. 自早期版本的Spark被重命名；在这种情况下，年龄较大 键名仍被接受，但优先级低于任何键名 较新的密钥的实例.

Any values specified as flags or in the properties file will be passed on to the application and merged with those specified through SparkConf. Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file. A few configuration keys have been renamed since earlier versions of Spark; in such cases, the older key names are still accepted, but take lower precedence than any instance of the newer key.

这篇关于应用程序配置文件中的setMaster与sparkSubmit上的--master标志之间存在火花差异或冲突的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！