在Windows上运行没有Admin权限的Hadoop MR作业 [英] Running Hadoop MR jobs without Admin privilege on Windows

问题描述

我在Windows中安装了Hadoop 2.3.0并能够成功执行MR作业。但是，当我尝试以正常特权执行MR作业（没有管理员权限）时，意味着作业失败，并出现以下异常。这里我尝试了一下Pig Script示例。

  2014-10-15 12:02:32,822 WARN [main] org.apache。 hadoop.security.UserGroupInformation：PriviledgedActionException as：kaveen（auth：SIMPLE）cause：java.io.IOException：Split class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit not found 
 2014-10- 15 12：02：32,823 WARN [main] org.apache.hadoop.mapred.YarnChild：异常正在运行child：java.io.IOException：Split类org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit未找到
 at org.apache.hadoop.mapred.MapTask.getSplitDetails（MapTask.java:362）
 at org.apache.hadoop.mapred.MapTask.runOldMapper（MapTask.java:403）
 at org.apache.hadoop.mapred.MapTask.run（MapTask.java:342）
 at org.apache.hadoop.mapred.YarnChild $ 2.run（YarnChild.java:168）
 at java.security .AccessController.doPrivileged（Native方法）
位于javax.security.auth.Subject.doAs（Subject.java:415）
位于org.apache.hadoop.se curl.UserGroupInformation.doAs（UserGroupInformation.java:1548）
 at org.apache.hadoop.mapred.YarnChild.main（YarnChild.java:163）
引起：java.lang.ClassNotFoundException：Class org .apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit在org.apache.hadoop.conf.Configuration.getClassByName（Configuration.java:1794）$ or $ $ b $ org.apache.hadoop处未找到
。 mapred.MapTask.getSplitDetails（MapTask.java:360）
 ... 7 more 
 
 2014-10-15 12:02:32,827 INFO [main] org.apache.hadoop.mapred .Task：Runnning清理任务
 2014-10-15 12:02:32,827 WARN [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter：输出路径在abortTask（）中为null 
  

更新：

我能够钻取问题，并发现在下面一行中引发的异常在MapTask.getSplitDetails（MapTask.java:363）方法中引发的异常。

 私人< T> T getSplitDetails（路径文件，长偏移量）
抛出IOException {
 FileSystem fs = file.getFileSystem（conf）; 
 FSDataInputStream inFile = fs.open（file）; 
 inFile.seek（offset）; 
 String className = StringInterner.weakIntern（Text.readString（inFile））; 
 Class< T> CLS; 
 try {
 cls =（Class< T>）conf.getClassByName（className）; 
} catch（ClassNotFoundException ce）{
 IOException wrap = new IOException（Split class+ className + 
not found）; 
 wrap.initCause（ce）; 
 
抛出包裹; 
 
 $ / code> 

但是如果我以管理员权限启动NodeManager发生。如果有人知道上述问题的原因和解决方案，我不知道为什么MR作业不能正常工作，当我启动具有普通特权的NodeManager时。

请尽快引导我。

使用下面的属性

可以更改hadoop的tmp目录位置的位置

p>

 <属性> 
< name> hadoop.tmp.dir< / name> 
< value> / other / tmp< / value> 
< / property> 
  

您的默认tmp位置是c：\ tmp，它需要管理员权限才能访问。将位置更改为任何子目录，然后尝试MR作业而无需管理员权限。

希望它有帮助。

I have installed Hadoop 2.3.0 in windows and able to execute MR jobs successfully. But when I trying to execute MR jobs in normal privilege (without admin privilege) means job get fails with following exception. Here I tried with Pig Script sample.

    2014-10-15 12:02:32,822 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:kaveen (auth:SIMPLE) cause:java.io.IOException: Split class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit not found
2014-10-15 12:02:32,823 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Split class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit not found
    at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:362)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:403)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1794)
    at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:360)
    ... 7 more

2014-10-15 12:02:32,827 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
2014-10-15 12:02:32,827 WARN [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Output Path is null in abortTask()

Update:

I was able to drill down the problem and found that the exception raised in the following line at method "MapTask.getSplitDetails(MapTask.java:363)".

private <T> T getSplitDetails(Path file, long offset) 
  throws IOException {
   FileSystem fs = file.getFileSystem(conf);
   FSDataInputStream inFile = fs.open(file);
   inFile.seek(offset);
   String className = StringInterner.weakIntern(Text.readString(inFile));
   Class<T> cls;
   try {
     cls = (Class<T>) conf.getClassByName(className);
   } catch (ClassNotFoundException ce) {
     IOException wrap = new IOException("Split class " + className + 
                                         " not found");
     wrap.initCause(ce);

     throw wrap;
   }

But If I start "NodeManager" with admin privilege mean the above exception won't occur. I don't know why MR job not working when I start "NodeManager" with normal privilege.

If anyone know the reason and solution for above problem. Please guide me as soon as possible.

解决方案

You can change the location of tmp directory location for hadoop using the below property

<property>
   <name>hadoop.tmp.dir</name>
   <value>/other/tmp</value>
</property>

Your default tmp location is c:\tmp which requires admin privilege to access. Change the location into any sub directory and try MR job without admin privilege.

Hope it helps.

这篇关于在Windows上运行没有Admin权限的Hadoop MR作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！