Hadoop IOException登录失败 [英] Hadoop IOException failure to login
问题描述
我对Hadoop很陌生.但是，我已经能够在服务器上的集群模式下使用Java 7成功设置hadoop 2.7.3.一切正常.
I'm pretty new to Hadoop. However, I've been able to successfully setup hadoop 2.7.3 with Java 7 in the cluster mode on my servers. Everything works totally fine.
但是，当我尝试切换至Java 8并启动dfs时，出现错误:
But then, when I try to switch to Java 8 and start dfs, there is an error:
Exception in thread "main" java.io.IOException: failure to login
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
... 6 more
似乎hadoop碰巧无法使用我在工作人员上的用户名登录.我尝试从其他来源查找它，并遵循它们的指导，例如添加环境变量HADOOP_USER_NAME，删除和重置整个datanode和namenode等.但是，在这种情况下，它们都不适合我.
It seems like hadoop happens to not be able to login using my username on the workers. I try to look it up from other sources and followed their guidances such as adding environment variable HADOOP_USER_NAME, deleting and reseting the whole datanode and namenode, etc. However, none of them works for me in this case.
关于这个问题的怪异之处是，当我尝试切换回Java 7时.Hadoop像以前一样对我来说工作正常.但是，我认为应该不会有问题，因为Java版本之间存在差异，因为Hadoop根据其文档与两者兼容.此外，群集中已经设置了受信任的ssh.
The weird thing about this problem is that when I try to switch back to Java 7. Hadoop works fine for me like before. However, I don't think there should be a problem because of the difference between the Java versions since Hadoop is compatible with both of them based on its documentation. Also, trusted ssh is already set up within the cluster.
下面是我的.bashrc配置文件:
Below is my .bashrc configurations fyi:
export JAVA_HOME="/scratch/dsat_server/jdk1.8.0_121"
export HADOOP_PREFIX="/scratch/dsat_server/hadoop-2.7.3"
export HADOOP_COMMON_LIB_NATIVE_DIR="$HADOOP_PREFIX/lib/native"
export HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=truei-Djava.library.path=$HADOOP_PREFIX/lib"
推荐答案
在使用Java 8从Docker容器运行hbase客户端时，我遇到了相同的问题.这显然是由使用本机的类com.sun.security.auth.module.UnixLoginModule引起的调用以获取Unix用户名.就我而言，它未在docker中映射，并且该类抛出NullPointerException.本身并不是hadoop中的错误.
I've encountered the same issue when running an hbase client from a docker container with Java 8. It is apparently caused by class com.sun.security.auth.module.UnixLoginModule which uses a native call to get the unix username. In my case, it is not mapped in docker, and the class throws a NullPointerException. It is not a bug in hadoop per se.
要指示hadoop绕过OS用户名的查找，我能够在所有初始化之前添加以下代码行:
To instruct hadoop to bypass the lookup of the OS username, I was able to add the following line of code before all initialization:
UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("hduser"));
在您的情况下，您正在运行服务器，因此注入代码的选项受到限制.相反，有两个选项:
In your case, you are running the server, so your options of injecting the code are limited. Instead there are two options:
- 改为尝试IBM JDK
- 尝试在工作程序($ whoami)上调试OS用户设置.如果显示无法找到用户ID XXXX的名称"之类的内容，请检查/etc/passwd设置
这篇关于Hadoop IOException登录失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
