使用python的CentOS流示例上的Hadoop - / mapred / local / taskTracker上的权限被拒绝 [英] Hadoop on CentOS streaming example with python - permission denied on /mapred/local/taskTracker

问题描述

我已经能够用python mapper& amp; amp;减速器。 mapred文件夹的位置是/ mapred / local / taskTracker
都是root&被映射的用户对该文件夹&子文件夹

然而，当我运行我的流时，它创建了映射但没有缩小，并给出以下错误
无法运行程序
/ mapred / local /的TaskTracker /根/ jobcache / job_201303071607_0035 / attempt_201303071607_0035_m_000001_3 /工作/./ mapper1.py
权限拒绝

我注意到，虽然它提供了一个+ RWX权限mapred /本地/ TaskTracker共同及其所有子目录，当MapReduce的创建适合这个工作的临时文件夹，文件夹没有为所有用户RWX ......因此我得到许可被拒绝错误

我一直在寻找这方面的论坛主题，虽然有线程提及相同的错误...我找不到任何解决方案。

任何帮助，将不胜感激。

解决方案

我假设你运行Hadoop守护进程作为用户根。在这种情况下，新创建文件的权限由用户 root 的 umask 确定。但是，您不得为 root 更改 umask 。

如果你想运行MapReduce作业和集群为不同的用户，这将是更好的启动Hadoop守护进程作为用户的hadoop 和MapReduce作业为用户映射精简。但是，两个用户都应该属于同一个组，即 hadoop 。此外，用户 hadoop 的 umask 应相应设置。

I have been able to set up the streaming example with python mapper & reducer. The mapred folder location is /mapred/local/taskTracker both root & mapred users have the ownership to this folder & sub folders

however when I run my streaming it creates maps but no reduces and gives the following error Cannot Run Program /mapred/local/taskTracker/root/jobcache/job_201303071607_0035/attempt_201303071607_0035_m_000001_3/work/./mapper1.py Permission Denied

I noticed that though it have provided a+rwx permission to mapred/local/taskTracker and all its sub directories, when mapreduce creates the temp folders for this job, the folders do not have the rwx for all users ...and hence I get the permission denied error

I have been looking for forum threads on this, and though there are threads mentioning the same error ...I could not find any responses with resolutions.

any help would be greatly appreciated

解决方案

I assume that you run your Hadoop daemons as user root. In this case the permissions of newly created files are determined by the umask of user root. However you must not change the umask for root.

If you'd like to run MapReduce jobs and cluster as different users, it would be better to start the Hadoop daemons as user hadoop and the MapReduce jobs as user mapreduce. However both users should belong to the same group, i.e. hadoop. Furthermore the umask for user hadoop shall be set accordingly.

这篇关于使用python的CentOS流示例上的Hadoop - / mapred / local / taskTracker上的权限被拒绝的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！