如何在Hadoop mapReduce中获得Kerberos而不是授权令牌？ [英] How to get Kerberos instead of delegation token in Hadoop mapReduce?

问题描述

我是Java用户，向Hadoop mapReduce提交作业时，它使用Kerberos为Hadoop进行身份验证，一旦成功，就会创建委托令牌并将作业提交传递给Hadoop，而不是Kerberos票据（为了安全原因如Hadoop所述）。现在，作业与我一样运行，但作业本身需要使用Kerberos向Hadoop之外的其他服务发送请求。现在我在Hadoop上没有kerberos TGT，我无法获得服务票证。

无论如何，我可以通过Kerberos票据工作吗？ （我知道这可能是危险的，因为我们不想传递秘密），JobConf可以将字符串传递给Hadoop的字符串对，但是我必须将TGT转换为json字符串并在作业运行期间还原它？ / p>

或者是否有可能使用授权令牌改革TGT？

我尝试过谷歌但没有太多信息，任何人都可以帮忙？谢谢。

** Editted： * *

看起来没有将TGT传递给Hadoop的简单方法，所以我将通过将作为字符串的TGT作为字符串通过作业配置映射传递给Hadoop来尝试以下方法（仅限字符串），并在Hadoop中运行作业时将字符串转换回TGT对象。我担心的是我将通过网络传递证书，这不是最佳实践，也是Hadoop没有通过Kerberos来确保安全的原因之一。如果我可以重新使用传递给Hadoop的重组TGT来获得服务票据，我会尽可能地加密TGT字符串以避免安全问题。

因此，在本地机器开始工作之前，代码将如下所示：

  import sun.security.krb5.Credentials; 
 
 Credentials tgt = Credentials.acquireTGTFromCache（null，null）; //确保在这个
 
之前完成kinit String tgtStr = tgt.convertToJsonString（）; //需要实现这个
 
 Job job = new Job（Test）; 
 JobConf jobConf = job.getJobConf（）; 
 jobConf.set（tgtStr，tgtStr）; 
 job.addTask（Test.class，run，null）; 
 job.submit（）; 
 job.waitForCompletion（true）; 
  

然后Hadoop运行的作业函数如下所示：

 配置conf = TaskContext.get（）。getConfiguration（）; 
 String tgtStr = conf.get（tgtStr）; 
 Credentials tgt = reformTGTFromString（tgtStr）; //需要实现这个
凭证serviceTicket = Credentials.acquireServiceCreds（servicePrincipal，tgt）; //这是得到任何服务票据
  

所以我需要实现两个函数来流TGT对象（ Credentials.class）字符串，然后将其重新回到对象。

任何人都知道更好的解决方案吗？通过反汇编Credentials字段并将它们转换为使用Base64编码器的字符串，形成一个JSON字符串并将其传递给Hadoop。

解析方法

使用RVM建议的配置映射或分布式缓存，然后重新构建运行在Hadoop上的作业中的Credentials对象，我可以取回Kerberos TGT并成功获取使用它的任何服务票据。所以这种方法是有效的，这里唯一需要非常谨慎的是通过网络传递的密钥的加密。

I'm a Java user, when submitting a job to Hadoop mapReduce, it uses Kerberos to authenticate for Hadoop, and upon success there's the delegation token created and passed with the job submission to Hadoop instead of the kerberos ticket (for security reason as stated by Hadoop). Now the job is running as me, but the job itself needs to use Kerberos to send request to other services outside Hadoop. Now I don't have kerberos TGT on Hadoop and I can't get the service ticket.

Is there anyway I can pass the Kerberos ticket with the job? (I know it might be dangerous since we don't want to pass the secret around), JobConf could pass the string to string pairs to Hadoop, but I have to convert the TGT to a json string and revert it during job running?

Or is it possible to use the delegation token reform TGT?

I tried to google it but not much information, anyone could help? Thank you.

**Editted:**

Looks like there's no easy way of doing this without passing the TGT to Hadoop, so I am going to try the following method by passing the TGT as string via job config map to Hadoop (String only), and convert the string back to TGT object when the job runs in Hadoop. The concern is I am going to pass the credentials over the network, which is not a best practice and one of the very reasons Hadoop didn't pass Kerberos around for security. If I could re-use the reformed TGT passed to Hadoop to get the service tickets, I will try to encrypt the TGT string as much as possible to avoid security issues.

So before starting a job in the local machine, the code would be like:

import sun.security.krb5.Credentials;  

Credentials tgt = Credentials.acquireTGTFromCache(null, null); // Make sure kinit is done before this

String tgtStr = tgt.convertToJsonString(); //Need to implement this

Job job = new Job("Test");
JobConf jobConf = job.getJobConf();
jobConf.set("tgtStr", tgtStr);
job.addTask(Test.class, "run", null);
job.submit();
job.waitForCompletion(true);

Then the function in the job for Hadoop to run would be like:

Configuration conf = TaskContext.get().getConfiguration();
String tgtStr = conf.get("tgtStr");
Credentials tgt = reformTGTFromString(tgtStr);//Need to implement this
Credentials serviceTicket = Credentials.acquireServiceCreds(servicePrincipal, tgt); //This is to get any service ticket

So I need to implement two function to stream TGT object (Credentials.class) to string and then reform it back to object.

Anyone knows a better solution for this? Thanks.

解决方案

By disassembling the Credentials fields and convert them to Strings using Base64 encoder, form a JSON string and pass it to Hadoop using config map or distributed cache suggested by RVM, and then reform the Credentials object in the job running on Hadoop, I can get back the Kerberos TGT and successfully get any service tickets using it. So this method works, and the only thing here needs to be very cautious is the encryption of the keys that are passed over network.

这篇关于如何在Hadoop mapReduce中获得Kerberos而不是授权令牌？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！