Hadoop WordCount示例卡住地图100％减少0％ [英] Hadoop WordCount example stuck at map 100% reduce 0%

问题描述

  [hadoop-1.0.2]→hadoop jar hadoop-examples-1.0.2.jar wordcount / user / abhinav / input / user / abhinav / output 
 Warning： $ HADOOP_HOME已弃用。 
 
 **** hdfs：// localhost：54310 / user / abhinav / input 
 12/04/15 15:52:31 INFO input.FileInputFormat：要输入的总输入路径： 1 
 12/04/15 15:52:31 WARN util.NativeCodeLoader：无法为您的平台加载native-hadoop库...在适用的情况下使用builtin-java类
 12/04/15 15 ：52：31 WARN snappy.LoadSnappy：Snappy本地库未加载
 12/04/15 15:52:31信息mapred.JobClient：正在运行的作业：job_201204151241_0010 
 12/04/15 15:52： 32信息mapred.JobClient：map 0％reduce 0％
 12/04/15 15:52:46信息mapred.JobClient：map 100％减少0％
  

我使用本指南在单个节点上设置了hadoop（http://www.michael-noll.com/tutorials/running-hadoop-on- ubuntu-linux-single-node-cluster /＃run-the-mapreduce-job），我试图运行一个提供的例子，但我陷入了地图100％减少0％。这可能是什么原因造成的？

首先，打开你的工作跟踪器，看看自由减速器插槽的数量，其他正在运行的作业 - 是否还有另一个正在运行的作业正在耗尽所有可用的减速机插槽。

一旦您向自己证明自己有一些可用的减速机插槽可用于运行减速机，请在作业跟踪器Web UI中找到您的工作，然后单击它打开它。您现在应该可以看到已完成的映射器的数量 - 确保这显示您没有正在运行的映射器。控制台中完成的百分比有时会存在，你可能有一个映射器，它正在提交它说它是100％，但有一个问题最终确定。

重新满足你所有的mappers已经完成，看看运行reducers的数量 - 这是否显示0？如果没有，则表明有一些正在运行 - 单击正在运行的reducer的数量以启动正在运行的reducers页面，现在单击一个实例，直到获得查看reducer日志的选项。您需要查看该减速器的所有日志（不是第一个/最后一个100k）。这应该告诉你你的reducer实际上在做什么 - 很可能试图将结果从映射器复制到reducer节点。我想这是你的问题，网络或磁盘空间之一，但无论如何，最终hadoop应该使reducer实例失效并重新安排它在另一个节点上运行。

[hadoop-1.0.2] → hadoop jar hadoop-examples-1.0.2.jar wordcount /user/abhinav/input     /user/abhinav/output
Warning: $HADOOP_HOME is deprecated.

****hdfs://localhost:54310/user/abhinav/input
12/04/15 15:52:31 INFO input.FileInputFormat: Total input paths to process : 1
12/04/15 15:52:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for     your platform... using builtin-java classes where applicable
12/04/15 15:52:31 WARN snappy.LoadSnappy: Snappy native library not loaded
12/04/15 15:52:31 INFO mapred.JobClient: Running job: job_201204151241_0010
12/04/15 15:52:32 INFO mapred.JobClient:  map 0% reduce 0%
12/04/15 15:52:46 INFO mapred.JobClient:  map 100% reduce 0%

I've set up hadoop on a single node using this guide (http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#run-the-mapreduce-job) and I'm trying to run a provided example but I'm getting stuck at map 100% reduce 0%. What could be causing this?

解决方案

First of all, open up your job tracker and look at the number of free reducer slots and other running jobs - is there another job running which is consuming all the free reducer slots when then become available.

Once you've proved to yourself that there are some free reducer slots available to run a reducer for you job, locate your job in the job tracker web ui and click on it to open it up. You should now be able to see the number of completed mappers - ensure this reads that you have no running mappers. The % complete in the console sometimes lies and you could have a mapper which is in the process of committing saying it's 100%, but having a problem finalizing.

Once you're satisfied that all your mappers have finished, look at the number of running reducers - does this show 0? If not does it show that some are running - click on the number of running reducers to bring up the running reducers page, now click through on an instance until you get an option to view the logs for the reducer. You'll want to view all the logs for this reducer (not the first / last 100k). This should tell you what your reducer is actually doing - most probably trying to copy the results from the mappers to the reducer node. I imagine this is where your problem is, one of network or disk space, but eitherway, eventually hadoop should fail the reducer instance out and reschedule it to run on another node.

这篇关于Hadoop WordCount示例卡住地图100％减少0％的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！