在Windows上使用Java和Kerberos Keytab访问Cloudera上的HDFS [英] Accessing HDFS on Cloudera with Java and Kerberos Keytab from Windows

问题描述

我试图连接到在Cloudera上运行的我的HDFS实例。我的第一步是启用Kerberos并创建Keytabs（如图所示这里）。

在接下来的步骤中，我想用密钥表进行身份验证。

 配置conf = new Configuration（）; 
 conf.set（fs.defaultFS，hdfs：// cloudera：8020）; 
 conf.set（hadoop.security.authentication，kerberos）; 
 
 UserGroupInformation.setConfiguration（conf）; 
 UserGroupInformation.loginUserFromKeytab（hdfs @ CLOUDERA，/etc/hadoop/conf/hdfs.keytab）; 
 
 FileSystem fs = FileSystem.get（conf）; 
 FileStatus [] fsStatus = fs.listStatus（new Path（/））; 
 for（int i = 0; i< fsStatus.length; i ++）{
 System.out.println（fsStatus [i] .getPath（）。toString（））; 
} 
  

失败，错误如下：

java.io.IOException：keytab中的hdfs @ CLOUDERA登录失败
/etc/hadoop/conf/hdfs.keytab：
javax.security.auth .login.LoginException：无法从用户获取密码

问题是：我如何正确处理keytab？我必须将其复制到我的本地机器吗？

解决方案

在Windows上运行Hadoop客户端以访问kerberized集群时，需要特定native library（即DLL）。

据我所知，没有什么好的理由，因为这个lib实际上并不是在一些自动回归测试之外使用的！）！所以这是Hadoop提交者对Hadoop用户造成的一个痛苦。

为了增加额外的痛苦，没有正式构建该DLL（以及Windows的存根，使它能够从Java中使用）。您必须自己从源代码中自行构建它 - 祝您好运 - 或者（em）（b）在互联网上搜索可下载的Hadoop-for-Windows运行时，并且祈祷不包含任何恶意软件。

最好的选择（对于64位Windows）在这里： https://github.com/steveloughran/winutils

...以及自述文件解释了为什么您可以合理地相信运行时。但是，如果你坚持使用一个较旧的32位Windows，那么你是自己的。

现在让我们假设你在Windows下的< br>
   C：\有些Dir \hadoop\bin\
（最后<$ c $

您必须将Hadoop客户端指向该运行时环境，有几个Java属性：

   - Dhadoop.home.dir = C：/ Some Dir / hadoop-Djava.library.path = C ：/ Some Dir / hadoop / bin


（注意Windows引用的双引号作为一个整体，以保护已翻译的路径中的嵌入空格（在Eclipse中，只是将这些道具放在虚拟参数下，包括引号）

现在，有Kerberos配置。如果您的KDC是您的公司Active Directory服务器，那么Java应自动查找配置参数。但是如果你的KDC在Linux上是一个独立的MIT Kerberos安装，那么你必须在集群上找到一个有效的 /etc/krb5.conf 文件，将它复制到你的Windows框中，并让Java将它与一个附加属性一起使用...

    - Djava.security.krb5.conf = C：/其他一些Dir / krb5 .conf

然后假设您已经在Linux机器上创建了您的keytab文件，使用 ktutil （或者一个Active Directory管理员用一些AD命令为你创建它），然后你将这个文件放在
   C：\ Some Some Dir\foo.keytab

在别的之前，如果keytab用于真正的Windows帐户（即您自己的帐户）或Prod服务帐户，则确保keytab是安全!! 使用Windows安全性对话框来限制对您帐户的访问权（也可能是System，用于备份）。因为该文件可以让任何人在任何机器上对群集（以及任何启用了Kerberos的系统，包括Windows）进行身份验证。

现在您可以尝试使用身份验证

UserGroupInformation.loginUserFromKeytab（[e-mail]，C：/ Some Other Dir / foo.keytab）;

如果不起作用，请使用环境变量启用Kerberos调试跟踪

   set HADOOP_JAAS_DEBUG = true code>

...以及Java属性

   -Dsun.security.krb5.debug = true

（在Eclipse中，分别在Environment和VM Arguments中设置它们）


I'm trying to connect to my HDFS instance running on Cloudera. My first step was enabling Kerberos and creating Keytabs (as shown here).

In the next step i would like to authenticate with a keytab.

Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://cloudera:8020");
conf.set("hadoop.security.authentication", "kerberos");

UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("hdfs@CLOUDERA", "/etc/hadoop/conf/hdfs.keytab");

FileSystem fs = FileSystem.get(conf);
FileStatus[] fsStatus = fs.listStatus(new Path("/"));
for (int i = 0; i < fsStatus.length; i++) {
    System.out.println(fsStatus[i].getPath().toString());
}

It fails with the following error

java.io.IOException: Login failure for hdfs@CLOUDERA from keytab /etc/hadoop/conf/hdfs.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

The question is: how do I correctly handle the keytab? Do i have to copy it to my local machine?

解决方案

When running a Hadoop client on Windows to reach a kerberized cluster, you need a specific "native library" (i.e. DLL).
As far as I can tell there is no good reason for that, because that lib is not actually used outside of some automated regression tests (!?!) so it's a pain inflicted to Hadoop users by Hadoop committers.

To add extra pain, there is no official build of that DLL (and of the Windows "stub" that enable its use from Java). You must either (a) build it yourself from source code -- good luck -- or (b) search the internet for a downloadable Hadoop-for-Windows runtime, and pray that is does not contain any malware.
The best option (for 64-bit Windows) is here: https://github.com/steveloughran/winutils
...and the ReadMe explains why you can reasonably trust that run-time. But if you are stuck with an older 32-bit Windows, then you are on your own.

Now let's assume you deployed that run-time on your Windows box under
  C:\Some Dir\hadoop\bin\
(the final bin is required; the embedded space is just extra fun)

You must point the Hadoop client to that run-time with a couple of Java properties:
  "-Dhadoop.home.dir=C:/Some Dir/hadoop" "-Djava.library.path=C:/Some Dir/hadoop/bin"
(note the double quotes around Windows args as a whole, to protect embedded spaces in the paths, which have been translated to Java style for extra fun)
(in Eclipse, just stuff these props under "VM Arguments", quotes included)

Now, there's the Kerberos config. If your KDC is your corporate Active Directory server, then Java should find the config parameters automatically. But if your KDC is a standalone "MIT Kerberos" install on Linux, then you have to find a valid /etc/krb5.conf file on the cluster, copy it on your Windows box, and have Java use it with an additional property...
  "-Djava.security.krb5.conf=C:/Some Other Dir/krb5.conf"

Then let's assume you have created your keytab file on a Linux box, using ktutil (or an Active Directory admin created it for you with some AD command) and you dropped the file under
  C:\Some Other Dir\foo.keytab
Before anything else, if the keytab is for a real Windows account -- i.e. your own account -- or a Prod service account, then make sure that keytab is secure!! Use the Windows Security dialog box to restrict access to your account only (and maybe System, for backups). Because that file could enable anyone, on any machine, to authenticate on the cluster (and any Kerberos-enabled system, including Windows).

Now you can try to authenticate using
UserGroupInformation.loginUserFromKeytab("[e-mail]", "C:/Some Other Dir/foo.keytab");

If it does not work, enable the Kerberos debug traces with both an environment variable
  set HADOOP_JAAS_DEBUG=true
...and a Java property
  -Dsun.security.krb5.debug=true
(in Eclipse, set these in "Environment" and "VM Arguments" respectively)

这篇关于在Windows上使用Java和Kerberos Keytab访问Cloudera上的HDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！