为嵌入在Java webapp中的客户端设置hadoop系统用户 [英] Set hadoop system user for client embedded in Java webapp

问题描述

我想将MapReduce作业从java web应用程序提交到远程Hadoop集群，但无法指定该作业应提交给哪个用户。我想配置和使用应该用于所有MapReduce作业的系统用户。

目前我无法指定任何用户，也不管hadoop作业运行的是什么在客户端系统的当前登录用户的用户名下。这会导致消息错误

 权限被拒绝：user = alice，access = WRITE，inode =staging：hduser： supergroup：rwxr-xr-x 
  

...其中alice是本地登录用户

我曾尝试过

1. 创建 UserGroupInformation 实例（包括代理和普通用户）和
   
2. 使用 -Duser.name = hduser ，改变 USER envar，并作为硬编码 System.setProperty（user.name，hduser ）通话。

...无济于事。关于1）我承认对这些类应该如何使用毫无头绪。另请注意，更改Java系统属性显然不是在Web应用程序中使用的真正解决方案。

是否有任何机构知道您如何指定哪个用户Hadoop用于连接到远程系统？

PS / Hadoop使用默认配置，这意味着在连接到群集时不使用身份验证，并且Kerberos没有用于与远程机器进行通信。

解决方案

最后，我偶然发现了常量

  static final String HADOOP_USER_NAME =HADOOP_USER_NAME;`
  

在 UserGroupInformation类中。

将其设置为环境变量，作为Java系统属性使用 System.setProperty（HADOOP_USER_NAME，hduser）; 启动Hadoop（使用 -D ）或编程方式使用任何你想要连接到远程Had的用户名oop集群。

I would like to submit MapReduce jobs from a java web application to a remote Hadoop cluster but am unable to specify which user the job should be submitted for. I would like to configure and use a system user which should be used for all MapReduce jobs.

Currently I am unable to specify any user and no matter what the hadoop job runs under the username of the currently logged in user of the client system. This causes an error with the message

Permission denied: user=alice, access=WRITE, inode="staging":hduser:supergroup:rwxr-xr-x

... where "alice" is the local, logged in user on the client machine.

I have tried

1. various combinations of creating UserGroupInformation instances (both proxies and normal user) and
2. setting the Java System property with -Duser.name=hduser, changing the USER envar and as a hard coded System.setProperty("user.name", "hduser") call.

... to no avail. Regarding 1) I admit to having no clue on how these classes are supposed to be used. Also please note that changing the Java System property is obviously not a real solution for use in the web application.

Does any body know how you specify which user Hadoop uses to connect to a remote system?

PS/ Hadoop is using the default configuration meaning that no authentication is used when connecting to the cluster and that Kerberos is not used in communicating with the remote machines.

解决方案

Finally I stumbled on the constant

static final String HADOOP_USER_NAME = "HADOOP_USER_NAME";`

in the UserGroupInformation class.

Setting this either as an environment variable, as a Java system property on startup (using -D) or programmatically with System.setProperty("HADOOP_USER_NAME", "hduser"); makes Hadoop use whatever username you want for connecting to the remote Hadoop cluster.

这篇关于为嵌入在Java webapp中的客户端设置hadoop系统用户的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！