spark中首选使用哪些配置选项? [英] Which configuration options are preferred in spark?

问题描述

我想询问在spark中优先考虑哪个配置选项?是配置文件还是我们在运行spark-submit shell时手动指定的选项? 如果我在配置文件中为执行程序内存提供了不同的选项，并且在运行spark-submit shell时指定了不同的值怎么办?

I wanted to enquire that which configuration option is given priority in spark? Is it the configuration file or the options we manually specify when running the spark-submit shell? What if I have different options for executor memory in my configuration file and I specify a different value while running the spark-submit shell?

推荐答案

Spark (1.5.0)

The Spark (1.5.0) configuration page clearly states what the priorities are:

直接在SparkConf上设置的属性具有最高优先级，然后将标志传递到spark-submit或spark-shell，然后是spark-defaults.conf文件中的选项.

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.

这是优先级顺序(从最高到最低):

So this is the priority order (from highest to lowest):

• 在 SparkConf 上设置的属性(在程序中).
• 将标记传递到spark-submit或spark-shell.
• 在spark-defaults.conf文件中设置的选项.

• Properties set on the SparkConf (in program).
• Flags passed to spark-submit or spark-shell.
• Options set in the spark-defaults.conf file.

这篇关于spark中首选使用哪些配置选项?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！