在hadoop上的FileNotFoundException [英] FileNotFoundException on hadoop

问题描述

MapReduce作业的sys输出日志文件在MapReduce作业的sys输出日志中，打印hashmap的内容。这表明它已经找到该文件，已经加载到数据结构中并执行了所需的操作。它遍历列表并打印其内容。因此，证明该操作是成功的。

然而，在运行MR作业几分钟后，我仍然遇到以下错误：

 
 13/01/27 18:44:21信息mapred.JobClient：任务ID：attempt_201301271841_0001_m_000001_2，状态：FAILED 
 java.io.FileNotFoundException：文件不存在：/ app / hadoop / jobs / nw_single_pred_in / predict $ b $ at org.apache.hadoop.hdfs.DFSClient $ DFSInputStream.openInfo（DFSClient.java:1843）
 at org.apache.hadoop.hdfs.DFSClient $ DFSInputStream。（DFSClient.java:1834）
 at org.apache.hadoop.hdfs.DFSClient.open（DFSClient.java:578）
 at org.apache.hadoop.hdfs.DistributedFileSystem.open（ 
 at org.apache.hadoop.fs.FileSystem.open（FileSystem.java:427）
 at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize（ LineRecordReader.java:67）
 at org.apache.hadoop.mapred.MapTask $ NewTrackingRecordReader.initialize（MapTask.java:522）
 at or g.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:763）
在org.apache.hadoop.mapred.MapTask.run（MapTask.java:370）
在org.apache。 hadoop.mapred.Child $ 4.run（Child.java:255）
 at java.security.AccessController.doPrivileged（Native Method）
 at javax.security.auth.Subject.doAs（Subject.java： 396）
在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1121）
 at org.apache.hadoop.mapred.Child.main（Child.java:249）
 
 

以下是将路径初始化为要放置在分布式缓存中的文件位置的部分

 
 
 //主内部，被try catch块包围，但这里没有抛出异常
 Configuration conf = new Configuration（）; 
 //其余与conf $ b $相关的内容b path pathfilefilepath = new Path（args [3]）; // args [3] = /app/hadoop/jobs/nw_single_pred_in/predict/knowledge.txt 
 DistributedCache.addCacheFile（knowledgefilepath.toUri（），conf）; 
 job.setJarByClass（NBprediction.class）; 
 //其余作业设置
 job.waitForCompletion（true）; //启动加载
 
 

这个是在map函数中的：

 
 
尝试{
 System.out.println（Inside try !!）; 
路径文件[] = DistributedCache.getLocalCacheFiles（context.getConfiguration（））; 
 Path cfile = new Path（files [0] .toString（））; //只有一个文件
 System.out.println（File path：+ cfile.toString（））; 
 CSVReader reader = new CSVReader（new FileReader（cfile.toString（）），'\t'）; （（nline = reader.readNext（））！= null）
 data.put（nline [0]，Double.parseDouble（nline [1]））; //加载到散列表
} 
 catch（Exception e）
 {//处理异常} 
 
 

帮助赞赏。

干杯！

解决方案

重新安装了hadoop并使用同一个jar来运行这个工作，问题就消失了。似乎是一个错误，而不是编程错误。

Inside my map function, I am trying to read a file from the distributedcache, load its contents into a hash map.

The sys output log of the MapReduce job prints the content of the hashmap. This shows that it has found the file, has loaded into the data structure and performed the needed operation. It iterates through the list and prints its contents. Thus proving that the operation was successful.

However, I still get the below error after a few minutes of running the MR job:

13/01/27 18:44:21 INFO mapred.JobClient: Task Id : attempt_201301271841_0001_m_000001_2, Status : FAILED
java.io.FileNotFoundException: File does not exist: /app/hadoop/jobs/nw_single_pred_in/predict
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1843)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1834)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:578)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

Here's the portion which initializes Path with the location of the file to be placed in the distributed cache

    // inside main, surrounded by try catch block, yet no exception thrown here
        Configuration conf = new Configuration();
        // rest of the stuff that relates to conf
        Path knowledgefilepath = new Path(args[3]); // args[3] = /app/hadoop/jobs/nw_single_pred_in/predict/knowledge.txt
        DistributedCache.addCacheFile(knowledgefilepath.toUri(), conf);
        job.setJarByClass(NBprediction.class);
        // rest of job settings 
        job.waitForCompletion(true); // kick off load

This one is inside the map function:

    try {
    System.out.println("Inside try !!");
    Path files[]= DistributedCache.getLocalCacheFiles(context.getConfiguration());
    Path cfile = new Path(files[0].toString()); // only one file
    System.out.println("File path : "+cfile.toString());
    CSVReader reader = new CSVReader(new FileReader(cfile.toString()),'\t');
    while ((nline=reader.readNext())!=null)
    data.put(nline[0],Double.parseDouble(nline[1])); // load into a hashmap
    }
    catch (Exception e)
    {// handle exception }

Help appreciated.

Cheers !

解决方案

Did a fresh installation of hadoop and ran the job with the same jar, the problem disappeared. Seems to be a bug rather than programming errors.

这篇关于在hadoop上的FileNotFoundException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！