从远程主机通过 Java API 访问 HDFS，用户认证 [英] HDFS access from remote host through Java API, user authentication
问题描述
我需要通过 Java API 从远程桌面使用 HDFS 集群.在写入访问权限之前，一切正常.如果我尝试创建任何文件，我会收到访问权限异常.路径看起来不错，但异常表明我的远程桌面用户名当然不是我访问所需 HDFS 目录所需的名称.
I need to use HDFS cluster from remote desktop through Java API. Everything works OK until it comes to write access. If I'm trying to create any file I receive access permission exception. Path looks good but exception indicates my remote desktop user name which is of course is not what I need to access needed HDFS directory.
问题是:- 有没有办法在 Java API 中使用简单"身份验证来表示不同的用户名?- 您能否用 Java API 示例对 hadoop/HDFS 中的身份验证/授权方案进行一些很好的解释?
The question is: - Is there any way to represent different user name using 'simple' authentication in Java API? - Could you please point some good explanation of authentication / authorization schemes in hadoop / HDFS preferable with Java API examples?
是的，我已经知道在这种情况下使用 shell 别名可能会重载whoami"，但我更愿意避免这样的解决方案.这里的另一个细节是我不喜欢使用一些技巧，比如通过 SSH 和脚本的管道.我想仅使用 Java API 执行所有操作.提前致谢.
Yes, I already know 'whoami' could be overloaded in this case using shell alias but I prefer to avoid solutions like this. Also specifics here is I dislike usage of some tricks like pipes through SSH and scripts. I'd like to perform everything using just Java API. Thank you in advance.
推荐答案
经过一番研究，我得出了以下解决方案:
After some studying I came to the following solution:
- 我实际上并不需要完整的 Kerberos 解决方案，目前客户端可以运行来自任何用户的 HDFS 请求就足够了.环境本身被认为是安全的.
- 这为我提供了基于 hadoop UserGroupInformation 类的解决方案.将来我可以扩展它以支持 Kerberos.
示例代码可能对人们进行假身份验证"和远程 HDFS 访问有用:
Sample code probably useful for people both for 'fake authentication' and remote HDFS access:
package org.myorg;
import java.security.PrivilegedExceptionAction;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;
public class HdfsTest {
public static void main(String args[]) {
try {
UserGroupInformation ugi
= UserGroupInformation.createRemoteUser("hbase");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://1.2.3.4:8020/user/hbase");
conf.set("hadoop.job.ugi", "hbase");
FileSystem fs = FileSystem.get(conf);
fs.createNewFile(new Path("/user/hbase/test"));
FileStatus[] status = fs.listStatus(new Path("/user/hbase"));
for(int i=0;i<status.length;i++){
System.out.println(status[i].getPath());
}
return null;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
}
对有类似问题的人有用的参考:
Useful reference for those who have a similar problem:
- Cloudera 博客文章Hadoop 中的授权和身份验证".简短，专注于对 hadoop 安全方法的简单解释.没有特定于 Java API 解决方案的信息，但有助于基本了解问题.
- Cloudera blog post "Authorization and Authentication In Hadoop". Short, focused on simple explanation of hadoop security approaches. No information specific to Java API solution but good for basic understanding of the problem.
更新:
对于那些在不需要本地用户的情况下使用命令行 hdfs 或 hadoop 实用程序的人的替代方案:
UPDATE:
Alternative for those who uses command line hdfs or hadoop utility without local user needed:
HADOOP_USER_NAME=hdfs hdfs fs -put /root/MyHadoop/file1.txt /
您实际上所做的是根据您的本地权限读取本地文件，但是当将文件放在 HDFS 上时，您会像用户 hdfs 一样进行身份验证.
What you actually do is you read local file in accordance to your local permissions but when placing file on HDFS you are authenticated like user hdfs.
这与所示的 API 代码具有非常相似的属性:
This has pretty similar properties to API code illustrated:
- 您不需要
sudo. - 您不需要真正合适的本地用户hdfs".
- 您无需复制任何内容或更改权限，因为前面几点.
这篇关于从远程主机通过 Java API 访问 HDFS，用户认证的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
