在不使用SPnego的情况下访问受Kerberos保护的WebHDFS [英] Accessing kerberos secured WebHDFS without SPnego

问题描述

我有一个使用WebHDFS管理HDFS的工作应用程序。
我需要能够在Kerberos安全集群上执行此操作。

问题是，没有库或扩展来协商我的票应用程序，我只有一个基本的HTTP客户端。

是否有可能创建一个Java服务来处理票据交换，它的应用程序在HTTP请求中使用？
换句话说，我的应用程序会要求Java服务协商票据，并且它会将服务票据以字符串或原始字符串的形式返回给我的应用程序，并且该应用程序会将其附加到HTTP请求中？

编辑：是否有类似雅致的解决方案，如@SamsonScharfrichter描述的HTTPfs？ （据我所知，它不支持委托代币）

编辑2：大家好，我还是完全迷失了。我试图找出没有任何运气的Hadoop-auth客户端。你能再帮我一次吗？我已经花了几个小时阅读它，但没有运气。
示例说明如下：

  * //建立初始连接
 * 
 * URL url =新的URL（http：// foo：8080 / bar）; 
 * AuthenticatedURL.Token token = new AuthenticatedURL.Token（）; 
 * AuthenticatedURL aUrl = new AuthenticatedURL（）; 
 * HttpURLConnection conn = new AuthenticatedURL（url，token）.openConnection（）; 
 * .... 
 * //使用'conn'实例
 * .... 
  

我已经在这里失去了。我需要什么样的初始连接？如何可以

  new AuthenticatedURL（url，token）.openConnection（）; 
  

取两个参数？这种情况下没有构造函数。 （因为这个，我得到错误）。不应该委托人指定某个地方？这可能不会是这么简单。

  URL url = new URL（http：//< host> ;: ？14000 / webhdfs / V1 / OP = liststatus）; 
 AuthenticatedURL.Token token = new AuthenticatedURL.Token（）; 
 
 HttpURLConnection conn = new AuthenticatedURL（url，token）.openConnection（url，token）; 
  

解决方案

使用Java代码和Hadoop Java API打开Kerberized会话，获取会话的委托令牌，并将该令牌传递给另一个应用 - 正如@tellisnz所建议的那样 - 有一个缺点：Java API需要相当多的依赖。大量的JAR以及Hadoop本地库）。如果你在Windows上运行你的应用程序，特别是它将是一个艰难的过程。

另外一个选择是使用Java代码加上WebHDFS来运行一个SPNEGOed查询， em> GET 委派令牌，然后将其传递给另一个应用程序 - 该选项要求服务器上绝对不存在Hadoop库。准系统版本将会像

  url urlGetToken = new URL（http：//< host>：< port> ; / webhdfs / v1 /？op = GETDELEGATIONTOKEN）; 
 HttpURLConnection cnxGetToken =（HttpURLConnection）urlGetToken.openConnection（）; 
 BufferedReader httpMessage = new BufferedReader（new InputStreamReader（cnxGetToken.getInputStream（）），1024）; 
模式regexHasToken = Pattern.compile（urlString [\：] +（。[^ \] +））; 
字符串httpMessageLine; （（httpMessageLine = httpMessage.readLine（））！= null）
 {Matcher regexToken = regexHasToken.matcher（httpMessageLine）; 
 if（regexToken.find（））
 {System.out.println（Use the template：http：//< Host>：< Port> / webhdfs / v1％AbsPath％？delegation =+ regexToken.group（1）+& op = ...）; } 
} 
 httpMessage.close（）; 
  

这就是我用来从Windows Powershell脚本（甚至是Excel宏）访问HDFS的方法。警告：在Windows中，您必须通过向JVM传递指向相应密钥表文件的JAAS配置来创建Kerberos TGT。但是这个警告也适用于Java API，无论如何。

I have a working application for managing HDFS using WebHDFS. I need to be able to do this on a Kerberos secured cluster.

The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client.

Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request? In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just attach it to the HTTP request?

EDIT: Is there a similar elegant solution like @SamsonScharfrichter described for HTTPfs? (To my knowledge, it does not support delegation tokens)

EDIT2: Hi guys, I am still completly lost. Im trying to figure out the Hadoop-auth client without any luck. Could you please help me out again? I already spent hours reading upon it without luck. The examples say to do this:

* // establishing an initial connection
*
* URL url = new URL("http://foo:8080/bar");
* AuthenticatedURL.Token token = new AuthenticatedURL.Token();
* AuthenticatedURL aUrl = new AuthenticatedURL();
* HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
* ....
* // use the 'conn' instance
* ....

Im lost already here. What initial connection do I need? How can

new AuthenticatedURL(url, token).openConnection();

take two parameters? there is no constructor for such a case. (im getting error because of this). Shouldnt a principal be somewhere specified? It is probably not going to be this easy.

    URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus");
    AuthenticatedURL.Token token = new AuthenticatedURL.Token();

    HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token);

解决方案

Using Java code plus the Hadoop Java API to open a Kerberized session, get the Delegation Token for the session, and pass that Token to the other app -- as suggested by @tellisnz -- has a drawback: the Java API requires quite a lot of dependencies (i.e. a lot of JARs, plus Hadoop native libraries). If you run you app on Windows, in particular, it will be a tough ride.

Another option is to use Java code plus WebHDFS to run a single SPNEGOed query and GET the Delegation Token, then pass it to the other app -- that option requires absolutely no Hadoop library on your server. The barebones version would be sthg like

URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ;
HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ;
BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ;
Pattern regexHasToken =Pattern.compile("urlString[\": ]+(.[^\" ]+)") ;
String httpMessageLine ;
while ( (httpMessageLine =httpMessage.readLine()) != null)
{ Matcher regexToken =regexHasToken.matcher(httpMessageLine) ;
  if (regexToken.find())
  { System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; }
}
httpMessage.close() ;

That's what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caveat: with Windows you have to create your Kerberos TGT on the fly, by passing to the JVM a JAAS config pointing to the appropriate keytab file. But that caveat also applies to the Java API, anyway.

这篇关于在不使用SPnego的情况下访问受Kerberos保护的WebHDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！