无需 SPnego 即可访问受 kerberos 保护的 WebHDFS [英] Accessing kerberos secured WebHDFS without SPnego

问题描述

我有一个使用 WebHDFS 管理 HDFS 的工作应用程序.我需要能够在 Kerberos 安全集群上执行此操作.

I have a working application for managing HDFS using WebHDFS. I need to be able to do this on a Kerberos secured cluster.

问题是，没有库或扩展可以为我的应用协商票证，我只有一个基本的 HTTP 客户端.

The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client.

是否可以创建一个 Java 服务来处理票证交换，一旦它获得服务票证，就可以将其传递给应用程序以在 HTTP 请求中使用?换句话说，我的应用程序会要求 Java 服务协商票证，它会以字符串或原始字符串的形式将服务票证返回给我的应用程序，应用程序会将其附加到 HTTP 请求?

Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request? In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just attach it to the HTTP request?

是否有类似为 HTTPfs 描述的@SamsonScharfrichter 的优雅解决方案?(据我所知，它不支持委托令牌)

Is there a similar elegant solution like @SamsonScharfrichter described for HTTPfs? (To my knowledge, it does not support delegation tokens)

大家好，我仍然完全迷失了.我试图在没有任何运气的情况下找出 Hadoop-auth 客户端.你能再帮我一次吗?我已经花了几个小时阅读它而没有运气.示例说要这样做:

Hi guys, I am still completly lost. Im trying to figure out the Hadoop-auth client without any luck. Could you please help me out again? I already spent hours reading upon it without luck. The examples say to do this:

* // establishing an initial connection
*
* URL url = new URL("http://foo:8080/bar");
* AuthenticatedURL.Token token = new AuthenticatedURL.Token();
* AuthenticatedURL aUrl = new AuthenticatedURL();
* HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
* ....
* // use the 'conn' instance
* ....

我已经在这里迷路了.我需要什么初始连接?怎么可能

Im lost already here. What initial connection do I need? How can

new AuthenticatedURL(url, token).openConnection();

取两个参数?这种情况没有构造函数.(我因此而出错).委托人不应该在某个地方指定吗?可能不会这么容易.

take two parameters? there is no constructor for such a case. (im getting error because of this). Shouldnt a principal be somewhere specified? It is probably not going to be this easy.

    URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus");
    AuthenticatedURL.Token token = new AuthenticatedURL.Token();

    HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token);

推荐答案

使用 Java 代码和 Hadoop Java API 打开 Kerberized 会话，获取会话的委托令牌，并将该令牌传递给另一个应用程序——作为@tellisnz 建议 - 有一个缺点:Java API 需要相当多的依赖项(即很多 JAR，加上 Hadoop 本机库).特别是，如果您在 Windows 上运行您的应用程序，那将是一段艰难的旅程.

Using Java code plus the Hadoop Java API to open a Kerberized session, get the Delegation Token for the session, and pass that Token to the other app -- as suggested by @tellisnz -- has a drawback: the Java API requires quite a lot of dependencies (i.e. a lot of JARs, plus Hadoop native libraries). If you run you app on Windows, in particular, it will be a tough ride.

另一种选择是使用 Java 代码和 WebHDFS 运行单个 SPNEGOed 查询并GET委托令牌，然后将其传递给另一个应用程序——该选项需要绝对不需要 Hadoop 库在您的服务器上.准系统版本会像

Another option is to use Java code plus WebHDFS to run a single SPNEGOed query and GET the Delegation Token, then pass it to the other app -- that option requires absolutely no Hadoop library on your server. The barebones version would be sthg like

URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ;
HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ;
BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ;
Pattern regexHasToken =Pattern.compile("urlString[": ]+(.[^" ]+)") ;
String httpMessageLine ;
while ( (httpMessageLine =httpMessage.readLine()) != null)
{ Matcher regexToken =regexHasToken.matcher(httpMessageLine) ;
  if (regexToken.find())
  { System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; }
}
httpMessage.close() ;

这就是我用来从 Windows Powershell 脚本(甚至是 Excel 宏)访问 HDFS 的方法.警告:对于 Windows，您必须动态创建 Kerberos TGT，方法是将指向适当密钥表文件的 JAAS 配置传递给 JVM.但无论如何，这个警告也适用于 Java API.

That's what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caveat: with Windows you have to create your Kerberos TGT on the fly, by passing to the JVM a JAAS config pointing to the appropriate keytab file. But that caveat also applies to the Java API, anyway.

这篇关于无需 SPnego 即可访问受 kerberos 保护的 WebHDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！