Hadoop文件中的分布式缓存未找到异常 [英] Distributed Caching in Hadoop File Not Found Exception
问题描述
它显示它创建了缓存文件。但是,当我去看看文件不存在的位置,当我尝试从我的映射器读取时,它显示File Not Found Exception(文件未找到异常)。
这是我要运行的代码:
JobConf conf2 = new JobConf(getConf(),CorpusCalculator.class);
conf2.setJobName(CorpusCalculator2);
//由reducer2发布的文件的分布式缓存在这里完成
conf2.addResource(新路径(/ opt / hadoop1 / conf / core-site.xml));
conf2.addResource(new Path(/ opt / hadoop1 / conf / hdfs-site.xml));
// cacheFile(conf2,new Path(outputPathofReducer2));
conf2.setNumReduceTasks(1);
//conf2.setOutputKeyComparatorClass()
conf2.setMapOutputKeyClass(FloatWritable.class);
conf2.setMapOutputValueClass(Text.class);
conf2.setOutputKeyClass(Text.class);
conf2.setOutputValueClass(Text.class);
conf2.setMapperClass(MapClass2.class);
conf2.setReducerClass(Reduce2.class);
FileInputFormat.setInputPaths(conf2,new Path(inputPathForMapper1));
FileOutputFormat.setOutputPath(conf2,new Path(outputPathofReducer3));
DistributedCache.addCacheFile(new Path(/ sunilFiles / M51.txt)。toUri(),conf2);
JobClient.runJob(conf
日志:
pre>
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager:在/ tmp1 / mapred / local / archive / -1731849462204707023_-2090562221_1263420527中创建M51.txt / localhost / sunilFiles-work-2204204368663038938 with rwxr-xr-x
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager:Cached /sunilFiles/M51.txt as / tmp1 / mapred / local / archive /-1731849462204707023_2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager:缓存/sunilFiles/M51.txt as / tmp1 / mapred / local /archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43:40 INFO mapred.JobClient:正在运行的工作:job_local_0003
13/04/27 04:43:40 INFO mapred.Task:Using ResourceCalculatorPlugin:o
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@8c2df1
13/04/27 04:43 :40 INFO mapred.MapT问:numReduceTasks:1
13/04/27 04:43:40 INFO mapred.MapTask:io.sort.mb = 100
13/04/27 04: 43:40 INFO mapred.MapTask:data buffer = 79691776/99614720
13/04/27 04:43:40 INFO mapred.MapTask:record buffer = 262144/327680
里面 configure()
:
异常读取DistribtuedCache:java.io.FileNotFoundException:/tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt(是一个目录)
Inside setup():/tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43 :41 WARN mapred.LocalJobRunner:job_local_0003
请帮助我,我一直在寻找解决方案持续6小时,明天我有作业提交。非常感谢你。
我通过使用copyMerge()属性解决了这个问题,它将所有存在的文件机器成一个单一的文件,该文件我已经成功地使用...如果我正在使用正常的文件它是失败的。谢谢你的回复。
It shows that it created cached files. But, when I go and look at the location the file is not present and when I am trying to read from my mapper it shows the File Not Found Exception.
This is the code that I am trying to run:
JobConf conf2 = new JobConf(getConf(), CorpusCalculator.class);
conf2.setJobName("CorpusCalculator2");
//Distributed Caching of the file emitted by the reducer2 is done here
conf2.addResource(new Path("/opt/hadoop1/conf/core-site.xml"));
conf2.addResource(new Path("/opt/hadoop1/conf/hdfs-site.xml"));
//cacheFile(conf2, new Path(outputPathofReducer2));
conf2.setNumReduceTasks(1);
//conf2.setOutputKeyComparatorClass()
conf2.setMapOutputKeyClass(FloatWritable.class);
conf2.setMapOutputValueClass(Text.class);
conf2.setOutputKeyClass(Text.class);
conf2.setOutputValueClass(Text.class);
conf2.setMapperClass(MapClass2.class);
conf2.setReducerClass(Reduce2.class);
FileInputFormat.setInputPaths(conf2, new Path(inputPathForMapper1));
FileOutputFormat.setOutputPath(conf2, new Path(outputPathofReducer3));
DistributedCache.addCacheFile(new Path("/sunilFiles/M51.txt").toUri(),conf2);
JobClient.runJob(conf
Logs:
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager: Creating M51.txt in /tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles-work-2204204368663038938 with rwxr-xr-x
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager: Cached /sunilFiles/M51.txt as /tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43:40 INFO filecache.TrackerDistributedCacheManager: Cached /sunilFiles/M51.txt as /tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43:40 INFO mapred.JobClient: Running job: job_local_0003
13/04/27 04:43:40 INFO mapred.Task: Using ResourceCalculatorPlugin : o
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@8c2df1
13/04/27 04:43:40 INFO mapred.MapTask: numReduceTasks: 1
13/04/27 04:43:40 INFO mapred.MapTask: io.sort.mb = 100
13/04/27 04:43:40 INFO mapred.MapTask: data buffer = 79691776/99614720
13/04/27 04:43:40 INFO mapred.MapTask: record buffer = 262144/327680
inside configure()
:
Exception reading DistribtuedCache: java.io.FileNotFoundException: /tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt (Is a directory)
Inside setup(): /tmp1/mapred/local/archive/-1731849462204707023_-2090562221_1263420527/localhost/sunilFiles/M51.txt
13/04/27 04:43:41 WARN mapred.LocalJobRunner: job_local_0003
Please help me out, I have been searching solution for this for last 6 hours continuously and tomorrow I have an assignment submission. Thank you very much.
I solved this problem by using copyMerge() Property which merges all the files that are present in various machines into a single file and that file I was successfully able to use..if I am using normal file it is failing. thanks for your replies guys.
这篇关于Hadoop文件中的分布式缓存未找到异常的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!