在Windows中使用MR2的NullPointerException [英] NullPointerException with MR2 in windows

问题描述

我在Windows中安装了Hadoop 2.3.0并能够成功执行MR作业。但是，在C＃中使用流示例进行尝试时，应用程序以下列异常结束： 16 18:21:06信息mapreduce.Job：任务ID：attempt_1400239892040_0003_r_000000_0，状态：FAILED
错误：java.lang.NullPointerException $ b $ org.apache.hadoop.mapred.Task.getFsStatistics（Task.java ：347）
at org.apache.hadoop.mapred.ReduceTask $ OldTrackingRecordWriter。< init>（ReduceTask.java:478）
at org.apache.hadoop.mapred.ReduceTask.runOldReducer（ReduceTask。 java：414）
at org.apache.hadoop.mapred.ReduceTask.run（ReduceTask.java:392）
at org.apache.hadoop.mapred.YarnChild $ 2.run（YarnChild.java:168 ）美元，java.security.AccessController.doPrivileged（本机方法）
在javax.security.auth.Subject.doAs（Subject.java:415）
在org.apache.hadoop.security b $ b .UserGroupInformation.doAs（UserGroupInformation.java:1548）
at org.apache.hadoop.mapred.YarnChild.main（YarnChild.java:163）

更新：

我能够深入了解问题，并发现在以下行中引发异常

  matchedStats = getFsStatistics（FileOutputFormat.getOutputPath（job），job）; 
 
 at org.apache.hadoop.mapred.ReduceTask $ OldTrackingRecordWriter。< init>（ReduceTask.java:478）
  

在上面，'FileOutputFormat.getOutputPath（job）'的结果返回null，它抛出空指针异常。以下是getOutputPath（）函数的代码。

  public static final String OUTDIR =mapreduce.output.fileoutputformat.outputdir; 
 
 public static Path getOutputPath（JobConf conf）{
 String name = conf.get（org.apache.hadoop.mapreduce.lib.output。
 FileOutputFormat.OUTDIR）; 
返回名称== null？ null：新路径（名称）; 
} 
  

因此，它是需要设置为属性值mapreduce.output.fileoutputformat。 outputdir在配置文件中的任何地方解决这个问题？

谢谢

解决方案

问题是hadoop服务已经从另一个用户[我的例子中的系统]开始，mapreduce样本是从我的本地用户提交的。因此，通过将[本地用户]的文件系统统计信息返回为空来解决问题。

一旦我从本地用户启动Hadoop，上述问题就得到解决。

I have installed Hadoop 2.3.0 in windows and able to execute MR jobs successfully. But while trying with streaming sample in C# [with HadoopSDK's .Net assemblies] the app ends with the following exception

14/05/16 18:21:06 INFO mapreduce.Job: Task Id : attempt_1400239892040_0003_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:478)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Update:

I was able to drill down the problem and found that the exception raised in the following line

 matchedStats = getFsStatistics(FileOutputFormat.getOutputPath(job), job);

at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:478)

In the above, the result of 'FileOutputFormat.getOutputPath(job)' returns null, which throws the null pointer exception. Below are the codes for the getOutputPath() function.

public static final String OUTDIR = "mapreduce.output.fileoutputformat.outputdir";

  public static Path getOutputPath(JobConf conf) {
  String name = conf.get(org.apache.hadoop.mapreduce.lib.output.
  FileOutputFormat.OUTDIR);
  return name == null ? null: new Path(name);
  }

So Is it needed to set value for property "mapreduce.output.fileoutputformat.outputdir" anywhere in the configuration files to get this issue resolved?

Thanks

解决方案

The problem is that the hadoop services have been started from a different user [SYSTEM in my case] and the mapreduce sample was submitted from my local user. So this makes the issue by returning the FileSystem statistics [for the local user] as Null.

Once i started Hadoop from my local user, the above issue gets resolved.

这篇关于在Windows中使用MR2的NullPointerException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！