Webhdfs上的权限错误 [英] Permissions error on webhdfs

问题描述

我正在使用Hadoop HDFS的REST接口作为通过网络存储文件的便捷方式。要测试我在我的Mac（10.8.5）上安装了hadoop，请按照以下说明操作：

http://importantfish.com/how-to-install-hadoop-on-mac-os-x/

这就像一个魅力，我可以启动hadoop并运行一个基本测试：

hadoop-examples-1.1 .2.jar pi 10 100

现在，我使用python客户端来处理webhdfs的HTTP请求：

http://pythonhosted.org/pywebhdfs/

但是当我尝试创建一个目录时，我遇到了一个基本的权限错误：

  from pywebhdfs.webhdfs import PyWebHdfsClient 
 hdfs = PyWebHdfsClient（）
 my_dir ='user / hdfs / data / new_dir'
 hdfs.make_dir（my_dir，权限= 755）
  

追溯（最近一次调用最后）：

文件，第1行，在

文件/Library/Python/2.7/site-packages/pywebhdfs/webhdfs.py，第207行，在make_dir

_raise_pywebhdfs_exception（response.status_code，response.text ）

文件/Library/Python/2.7/site-packages/pywebhdfs/webhdfs.py，第428行，在_raise_pywebhdfs_exception

中引发errors.PyWebHdfsException（msg = message）<
pywebhdfs.errors.PyWebHdfsException：{RemoteException：{exception：AccessControlException，javaClassName：org.apache.hadoop.security.AccessControlException，message：Permission denied： user = webuser，access = WRITE，inode = \user\：mlmiller：supergroup：rwxr-xr-x}} 我也尝试过将用户指定为'hdfs'而不是python lib的'webhdfs'，但得到相同的结果。读完30分钟后，我放弃了，并意识到我不理解hdfs用户，hadoop安全性（我在安装isntructions后启用）和我的unix用户和权限之间的相互作用。

解决方案

您需要让PyWebHdfsClient用户名匹配一个对您尝试写入的目录具有权限的unix用户。启动namenode服务的用户默认为超级用户。

我编写了用于响应工作需要的pywebhdfs客户端。如果您有任何问题或想要询问客户端的功能，请在github上留下问题，我可以解决它。

https://github.com/ProjectMeniscus/pywebhdfs/issues

谢谢你

I'm working on using the REST interface to Hadoop's HDFS as a convenient way to store files over the network. To test I installed hadoop on my mac (10.8.5) following these instructions:

http://importantfish.com/how-to-install-hadoop-on-mac-os-x/

That worked like a charm and I'm able to start hadoop and run a basic test:

hadoop-examples-1.1.2.jar pi 10 100

Now, I'm using the python client to handle the HTTP requests to/from webhdfs:

http://pythonhosted.org/pywebhdfs/

But I'm stumbling on a basic permissions error when I try to create a directory:

from pywebhdfs.webhdfs import PyWebHdfsClient  
hdfs = PyWebHdfsClient()  
my_dir = 'user/hdfs/data/new_dir'  
hdfs.make_dir(my_dir, permission=755)  

Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/pywebhdfs/webhdfs.py", line 207, in make_dir
_raise_pywebhdfs_exception(response.status_code, response.text)
File "/Library/Python/2.7/site-packages/pywebhdfs/webhdfs.py", line 428, in _raise_pywebhdfs_exception
raise errors.PyWebHdfsException(msg=message)
pywebhdfs.errors.PyWebHdfsException: {"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=webuser, access=WRITE, inode=\"user\":mlmiller:supergroup:rwxr-xr-x"}}

I've also tried specifying the user as 'hdfs' instead of the python lib's defeault to 'webhdfs' but get the same result. After 30 minutes reading I gave up and realized I don't understand the interplay of hdfs users, hadoop security (which I enabled following the install isntructions) and my unix user and permissions.

解决方案

You need to have the PyWebHdfsClient user_name match a unix user that has permission to the directory you are trying to write to. The user that starts the namenode service is by default the "superuser"

I wrote the pywebhdfs client you are using in response to a need at work. If you have any issues or would like to ask for features on the client itself please leave an issue on github and I can address it.

https://github.com/ProjectMeniscus/pywebhdfs/issues

Thank you

这篇关于Webhdfs上的权限错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！