Hadoop WordCount 示例卡在地图 100% 减少 0% [英] Hadoop WordCount example stuck at map 100% reduce 0%

问题描述

[hadoop-1.0.2] → hadoop jar hadoop-examples-1.0.2.jar wordcount /user/abhinav/input     /user/abhinav/output
Warning: $HADOOP_HOME is deprecated.

****hdfs://localhost:54310/user/abhinav/input
12/04/15 15:52:31 INFO input.FileInputFormat: Total input paths to process : 1
12/04/15 15:52:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for     your platform... using builtin-java classes where applicable
12/04/15 15:52:31 WARN snappy.LoadSnappy: Snappy native library not loaded
12/04/15 15:52:31 INFO mapred.JobClient: Running job: job_201204151241_0010
12/04/15 15:52:32 INFO mapred.JobClient:  map 0% reduce 0%
12/04/15 15:52:46 INFO mapred.JobClient:  map 100% reduce 0%

我已经使用本指南在单个节点上设置了 hadoop (http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#run-the-mapreduce-job) 并且我正在尝试运行提供的示例，但我卡在 map 100% reduce 0% 上.这可能是什么原因造成的?

I've set up hadoop on a single node using this guide (http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#run-the-mapreduce-job) and I'm trying to run a provided example but I'm getting stuck at map 100% reduce 0%. What could be causing this?

推荐答案

首先，打开您的作业跟踪器并查看空闲的减速器插槽和其他正在运行的作业的数量 - 是否有另一个正在运行的作业正在消耗所有空闲减速器插槽可用时.

First of all, open up your job tracker and look at the number of free reducer slots and other running jobs - is there another job running which is consuming all the free reducer slots when then become available.

一旦您向自己证明有一些免费的减速器插槽可用于为您的作业运行减速器，请在作业跟踪器 Web ui 中找到您的作业并单击它以将其打开.您现在应该能够看到已完成的映射器的数量 - 确保它读取您没有正在运行的映射器.控制台中的完成百分比有时是谎言，您可能有一个正在提交的映射器说它是 100%，但在最终确定时遇到问题.

Once you've proved to yourself that there are some free reducer slots available to run a reducer for you job, locate your job in the job tracker web ui and click on it to open it up. You should now be able to see the number of completed mappers - ensure this reads that you have no running mappers. The % complete in the console sometimes lies and you could have a mapper which is in the process of committing saying it's 100%, but having a problem finalizing.

一旦您对所有映射器都已完成感到满意，请查看正在运行的减速器的数量 - 这是否显示为 0?如果不是，它不显示一些正在运行 - 单击正在运行的减速器的数量以打开正在运行的减速器页面，现在单击一个实例，直到您获得查看减速器日志的选项.您需要查看此减速器的所有日志(不是第一个/最后一个 100k).这应该会告诉您您的减速器实际在做什么 - 很可能是尝试将结果从映射器复制到减速器节点.我想这就是您的问题所在，网络或磁盘空间之一，但无论如何，最终 hadoop 应该使 reducer 实例失败并重新安排它在另一个节点上运行.

Once you're satisfied that all your mappers have finished, look at the number of running reducers - does this show 0? If not does it show that some are running - click on the number of running reducers to bring up the running reducers page, now click through on an instance until you get an option to view the logs for the reducer. You'll want to view all the logs for this reducer (not the first / last 100k). This should tell you what your reducer is actually doing - most probably trying to copy the results from the mappers to the reducer node. I imagine this is where your problem is, one of network or disk space, but eitherway, eventually hadoop should fail the reducer instance out and reschedule it to run on another node.

这篇关于Hadoop WordCount 示例卡在地图 100% 减少 0%的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！