为嵌入在 Java webapp 中的客户端设置 hadoop 系统用户 [英] Set hadoop system user for client embedded in Java webapp

问题描述

我想将 MapReduce 作业从 Java Web 应用程序提交到远程 Hadoop 集群，但无法指定应向哪个用户提交作业.我想配置和使用一个应该用于所有 MapReduce 作业的系统用户.

I would like to submit MapReduce jobs from a java web application to a remote Hadoop cluster but am unable to specify which user the job should be submitted for. I would like to configure and use a system user which should be used for all MapReduce jobs.

目前我无法指定任何用户，无论 hadoop 作业在客户端系统当前登录用户的用户名下运行什么.这会导致消息错误

Currently I am unable to specify any user and no matter what the hadoop job runs under the username of the currently logged in user of the client system. This causes an error with the message

Permission denied: user=alice, access=WRITE, inode="staging":hduser:supergroup:rwxr-xr-x

...其中alice"是客户端机器上的本地登录用户.

... where "alice" is the local, logged in user on the client machine.

我试过了

1. 创建UserGroupInformation 实例(代理和普通用户)和
的各种组合
2. 使用 -Duser.name=hduser 设置 Java System 属性，更改 USER 环境并将其作为硬编码的 System.setProperty("user.name", "hduser") 调用.

1. various combinations of creating UserGroupInformation instances (both proxies and normal user) and
2. setting the Java System property with -Duser.name=hduser, changing the USER envar and as a hard coded System.setProperty("user.name", "hduser") call.

……无济于事.关于 1) 我承认不知道应该如何使用这些类.另请注意，更改 Java System 属性显然不是在 Web 应用程序中使用的真正解决方案.

... to no avail. Regarding 1) I admit to having no clue on how these classes are supposed to be used. Also please note that changing the Java System property is obviously not a real solution for use in the web application.

有谁知道您如何指定 Hadoop 使用哪个用户连接到远程系统?

PS/Hadoop 使用默认配置，这意味着在连接到集群时不使用身份验证，并且在与远程机器通信时不使用 Kerberos.

PS/ Hadoop is using the default configuration meaning that no authentication is used when connecting to the cluster and that Kerberos is not used in communicating with the remote machines.

推荐答案

最后我偶然发现了常量

static final String HADOOP_USER_NAME = "HADOOP_USER_NAME";`

在UserGroupInformation 类中.

将其设置为环境变量、启动时的 Java 系统属性(使用 -D)或使用 System.setProperty("HADOOP_USER_NAME", "hduser") 以编程方式设置； 使 Hadoop 使用您想要的任何用户名连接到远程 Hadoop 集群.

Setting this either as an environment variable, as a Java system property on startup (using -D) or programmatically with System.setProperty("HADOOP_USER_NAME", "hduser"); makes Hadoop use whatever username you want for connecting to the remote Hadoop cluster.

这篇关于为嵌入在 Java webapp 中的客户端设置 hadoop 系统用户的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！