如何通过特定用户初始化Spark Shell以通过Apache Spark将数据保存到hdfs [英] how to Intialize the spark shell with a specific user to save data to hdfs by apache spark

问题描述

• 使用ubuntu即时通讯
• 使用intellij使用火花依赖
• 找不到命令"spark"，但可以使用以下命令安装:..(当我在shell中输入spark时)
• 我有两个用户amine和hadoop_amine(已设置hadoop hdfs)

• im using ubuntu
• im using spark dependency using intellij
• Command 'spark' not found, but can be installed with: .. (when i enter spark in shell)
• i have two user amine , and hadoop_amine (where hadoop hdfs is set)

当我尝试将数据帧保存到HDFS(火花scala)时:

when i try to save a dataframe to HDFS (spark scala):

procesed.write.format("json").save("hdfs://localhost:54310/mydata/enedis/POC/processed.json")

我遇到了这个错误

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/mydata/enedis/POC":hadoop_amine:supergroup:drwxr-xr-x

推荐答案

尝试更改HDFS目录的权限或仅更改您的spark用户！ 要更改目录权限，您可以像这样使用hdfs命令行

Try to change the permissions of the HDFS directory or change your spark user simply! For changing the directory permission you can use hdfs command line like this

hdfs dfs -chmod  ...

在spark-submit中，您可以使用proxy-user选项 最后，您可以像以下命令那样以适当的用户身份运行spark-submit或spark-shell:

In spark-submit you can use the proxy-user option And at last, you can run the spark-submit or spark-shell with the proper user like this command:

sudo -u hadoop_amine spark-submit ...

这篇关于如何通过特定用户初始化Spark Shell以通过Apache Spark将数据保存到hdfs的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！