如何更改SparkContext.sparkUser()设置(在pyspark中)? [英] How can I change SparkContext.sparkUser() setting (in pyspark)?

问题描述

我是Spark和pyspark的新手.
我使用pyspark，在处理rdd之后，我尝试使用saveAsTextfile()函数将其保存到hdfs. 但是我收到"权限被拒绝"错误消息，因为pyspark尝试编写hdfs 使用我的本地帐户"kjlee"，该帐户在hdfs系统上不存在.

I am new with Spark and pyspark.
I use pyspark, after my rdd processing, I tried to save it to hdfs using the saveAsTextfile() function. But I get a 'permission denied' error message because pyspark tries to write hdfs using my local account, 'kjlee', which does not exist on the hdfs system.

我可以通过SparkContext().sparkUser()检查spark用户名，但是找不到如何更改spark用户名.

I can check the spark user name by SparkContext().sparkUser(), But I can't find how to change the spark user name.

如何更改Spark用户名?

How can I change the spark user name?

推荐答案

对此有一个环境变量:HADOOP_USER_NAME 因此只需使用export HADOOP_USER_NAME=anyuser或在pyspark中就可以使用os.environ["HADOOP_USER_NAME"] = "anyuser"

There is an environment variable for this : HADOOP_USER_NAME so simply use export HADOOP_USER_NAME=anyuser or in pyspark you can use os.environ["HADOOP_USER_NAME"] = "anyuser"

这篇关于如何更改SparkContext.sparkUser()设置(在pyspark中)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！