如何从pyspark设置hadoop配置值 [英] How to set hadoop configuration values from pyspark

问题描述

SparkContext的Scala版本具有属性

The Scala version of SparkContext has the property

sc.hadoopConfiguration

我已经成功地使用它来设置Hadoop属性(在Scala中)

I have successfully used that to set Hadoop properties (in Scala)

例如

sc.hadoopConfiguration.set("my.mapreduce.setting","someVal")

但是，SparkContext的python版本缺少该访问器.有什么方法可以将Hadoop配置值设置到PySpark上下文使用的Hadoop配置中?

However the python version of SparkContext lacks that accessor. Is there any way to set Hadoop configuration values into the Hadoop Configuration used by the PySpark context?

推荐答案

sc._jsc.hadoopConfiguration().set('my.mapreduce.setting', 'someVal')

应该工作

这篇关于如何从pyspark设置hadoop配置值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！