如果纱线（MRv2）或贴图（MRv1）以外的用户提交Hadoop作业失败， [英] Hadoop jobs fail when submitted by users other than yarn (MRv2) or mapred (MRv1)

问题描述

我正在运行一个运行MRv1（CDH5）与LocalFileSystem配对的测试集群，并且我能够以mapred方式运行作业的唯一用户（如mapred是启动jobtracker / tasktracker守护程序的用户）。当以其他用户的身份提交作业时，作业失败，因为jobtracker / tasktracker无法在.staging目录下找到job.jar。

我有完全相同的在与LocalFileSystem配对时，即在由'yarn'以外的用户提交作业时，应用程序主文件无法在.staging目录下找到job.jar，从而与YARN（MRv2）出现问题。

在检查提交作业的用户的.staging目录时，我发现job.jar存在于.staging目录下，但.staging目录中的权限设置为700（drwx-- ----），因此应用程序主/任务跟踪器无法访问job.jar和支持文件。

我们使用LocalFileSystem运行测试集群，因为我们

在这方面的任何帮助都非常有帮助。

解方案

这个工作对我来说，我只设置在MR V1此属性：

 <性> 
< name> hadoop.security.authorization< / name> 
<值>简单< /值> 
< / property> 
  

请仔细阅读：

存取控制列表
$ {HADOOP_CONF_DIR} /hadoop-policy.xml为每个Hadoop服务定义一个访问控制列表。每个访问控制列表都有一个简单的格式：

用户和组列表都是用逗号分隔的名称列表。这两个列表以空格分隔。

示例：user1，user2 group1，group2。

添加如果只提供一个组列表，则在该行的开始处留空，相当于以comman分隔的用户列表后跟一个空格或不包含任何给定用户。

特殊值*表示允许所有用户访问该服务。

刷新服务级别授权配置
服务级别授权可以更改NameNode和JobTracker的配置，而无需重新启动任一Hadoop主守护程序。群集管理员可以在主节点上更改$ {HADOOP_CONF_DIR} /hadoop-policy.xml，并指示NameNode和JobTracker分别通过-refreshServiceAcl开关将其各自的配置重新加载到dfsadmin和mradmin命令。

刷新NameNode的服务级授权配置：

$ bin / hadoop dfsadmin -refreshServiceAcl

刷新JobTracker的服务级别授权配置：

$ bin / hadoop mradmin -refreshServiceAcl

当然，可以使用$ {HADOOP_CONF_DIR} /hadoop-policy.xml中的security.refresh.policy.protocol.acl属性来限制对某些用户/组刷新服务级别授权配置的权限。

示例
仅允许mapreduce组中的用户alice，bob和用户向MapReduce集群提交作业：
$ b $ b

 <属性> 
< name> security.job.submission.protocol.acl< / name> 
< value>爱丽丝，bob mapreduce< /值> 
< / property> 
  

只允许DataNode作为属于组数据节点的用户与NameNode进行通信：

 <属性> 
< name> security.datanode.protocol.acl< / name> 
<值> datanodes< /值> 
< / property> 
允许任何用户作为DFSClient与HDFS集群对话：
 
<属性> 
< name> security.client.protocol.acl< / name> 
<值> *< /值> 
< / property> 
  

I am running a test cluster running MRv1 (CDH5) paired with LocalFileSystem, and the only user I am able to run jobs as is mapred (as mapred is the user starting the jobtracker/tasktracker daemons). When submitting jobs as any other user, the jobs fail because the jobtracker/tasktracker is unable to find the job.jar under the .staging directory.

I have the exact same issue with YARN (MRv2) when paired with LocalFileSystem, i.e. when submitting jobs by a user other than 'yarn', the application master is unable to locate the job.jar under the .staging directory.

Upon inspecting the .staging directory of the user submitting the job I found that job.jar exists under the .staging// directory, but the permissions on the and .staging directories are set to 700 (drwx------) and hence the application master / tasktracker is not able to access the job.jar and supporting files.

We are running the test cluster with LocalFileSystem since we use only MapReduce part of the Hadoop project paired with OCFS in our production setup.

Any assistance in this regard would be immensely helpful.

解决方案

This worked for me, I just set this property in MR v1:

<property>
    <name>hadoop.security.authorization</name>
    <value>simple</value>
  </property>

Please go through this:

Access Control Lists ${HADOOP_CONF_DIR}/hadoop-policy.xml defines an access control list for each Hadoop service. Every access control list has a simple format:

The list of users and groups are both comma separated list of names. The two lists are separated by a space.

Example: user1,user2 group1,group2.

Add a blank at the beginning of the line if only a list of groups is to be provided, equivalently a comman-separated list of users followed by a space or nothing implies only a set of given users.

A special value of * implies that all users are allowed to access the service.

Refreshing Service Level Authorization Configuration The service-level authorization configuration for the NameNode and JobTracker can be changed without restarting either of the Hadoop master daemons. The cluster administrator can change ${HADOOP_CONF_DIR}/hadoop-policy.xml on the master nodes and instruct the NameNode and JobTracker to reload their respective configurations via the -refreshServiceAcl switch to dfsadmin and mradmin commands respectively.

Refresh the service-level authorization configuration for the NameNode:

$ bin/hadoop dfsadmin -refreshServiceAcl

Refresh the service-level authorization configuration for the JobTracker:

$ bin/hadoop mradmin -refreshServiceAcl

Of course, one can use the security.refresh.policy.protocol.acl property in ${HADOOP_CONF_DIR}/hadoop-policy.xml to restrict access to the ability to refresh the service-level authorization configuration to certain users/groups.

Examples Allow only users alice, bob and users in the mapreduce group to submit jobs to the MapReduce cluster:

<property>
     <name>security.job.submission.protocol.acl</name>
     <value>alice,bob mapreduce</value>
</property>

Allow only DataNodes running as the users who belong to the group datanodes to communicate with the NameNode:

<property>
     <name>security.datanode.protocol.acl</name>
     <value>datanodes</value>
</property>
Allow any user to talk to the HDFS cluster as a DFSClient:

<property>
     <name>security.client.protocol.acl</name>
     <value>*</value>
</property>

这篇关于如果纱线（MRv2）或贴图（MRv1）以外的用户提交Hadoop作业失败，的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！