spark 2.1.0 会话配置设置 (pyspark) [英] spark 2.1.0 session config settings (pyspark)
问题描述
我正在尝试覆盖 spark 会话/spark 上下文默认配置,但它正在选择整个节点/集群资源.
I am trying to overwrite the spark session/spark context default configs, but it is picking entire node/cluster resource.
spark = SparkSession.builder
.master("ip")
.enableHiveSupport()
.getOrCreate()
spark.conf.set("spark.executor.memory", '8g')
spark.conf.set('spark.executor.cores', '3')
spark.conf.set('spark.cores.max', '3')
spark.conf.set("spark.driver.memory",'8g')
sc = spark.sparkContext
当我将配置放入 spark submit 时它工作正常
It works fine when i put the configuration in spark submit
spark-submit --master ip --executor-cores=3 --diver 10G code.py
推荐答案
您实际上并未使用此代码覆盖任何内容.只是为了让您亲眼看看,请尝试以下操作.
You aren't actually overwriting anything with this code. Just so you can see for yourself try the following.
一旦你启动 pyspark shell 类型:
As soon as you start pyspark shell type:
sc.getConf().getAll()
这将显示所有当前的配置设置.然后尝试您的代码并再次执行.没有任何变化.
This will show you all of the current config settings. Then try your code and do it again. Nothing changes.
您应该做的是创建一个新配置并使用它来创建 SparkContext.这样做:
What you should do instead is create a new configuration and use that to create a SparkContext. Do it like this:
conf = pyspark.SparkConf().setAll([('spark.executor.memory', '8g'), ('spark.executor.cores', '3'), ('spark.cores.max', '3'), ('spark.driver.memory','8g')])
sc.stop()
sc = pyspark.SparkContext(conf=conf)
然后你可以像上面一样检查自己:
Then you can check yourself just like above with:
sc.getConf().getAll()
这应该反映您想要的配置.
This should reflect the configuration you wanted.
这篇关于spark 2.1.0 会话配置设置 (pyspark)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!