Apache的Hadoop的地图reduce任务冰点 [英] Apache hadoop map reduce task freezing

问题描述

在我的设置Hadoop的多节点集群我也跑了著名的单词计数图减少例子。但我没有得到任何输出，但冻结的任务。在这里，我得到什么。

After I setup hadoop multi node cluster I did ran famous wordcount map reduce example. But I didnt get any output but freezing the task. Here What I get..

12/09/12 13:01:29 INFO input.FileInputFormat: Total input paths to process : 3
12/09/12 13:01:29 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/12 13:01:29 WARN snappy.LoadSnappy: Snappy native library not loaded
12/09/12 13:01:30 INFO mapred.JobClient: Running job: job_201209121300_0002
12/09/12 13:01:31 INFO mapred.JobClient:  map 0% reduce 0%
12/09/12 13:01:45 INFO mapred.JobClient:  map 33% reduce 0%
12/09/12 13:01:48 INFO mapred.JobClient:  map 100% reduce 0%
12/09/12 13:01:54 INFO mapred.JobClient:  map 100% reduce 11%

但也有在主机和从机日志没有例外。但是，在奴隶任务跟踪以下日志打印continously。

But there are no exceptions in logs in both master and slave. But in slaves task tracker following logs print continously.

2012-09-12 13:23:14,573 INFO org.apache.hadoop.mapred.TaskTracker:     attempt_201209121300_0002_r_000000_0 0.11111112% reduce > copy (1 of 3 at 0.04 MB/s) >

这是我没有配置Hadoop的一个节点，运行上面的任务之前，我也得到了成功的输出。

Before this I did configure hadoop single node, run the above task and I get the successful output.

P.S：
1.我有两个节点和工作的主从。两个IP地址在/ etc /中两个节点的主机。

2.可以SSH到每个主机和从机没有密码。 （密码登录）

3.After我在主机上运行start-dfs.sh也把本机（我JPS选中）
运行
4.这里是我按照教程。
的http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

在两台机器5.Firewall禁用。

p.s : 1.I have two nodes and work as master and slave. Both ip addresses are in /etc/hosts in both nodes.
2.can ssh to each master and slave without password. (passwordless login)
3.After I run the start-dfs.sh in master it also run in the slave machine (I checked with jps)
4. Here is the tutorial that I follow. http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
5.Firewall disabled in both machines.

我怎样才能解决这个问题？

How can I resolve this issue ?

推荐答案

最后我做到了。这里是我我所做的。我检查了运行图减少作业时的错误日志。每个工作Hadoop的产生，在日志/ userlogs / JOB_ID / *找到一些作业日志。所以，当我检查这些日志，我终于可以看到一些异常来知道的的UnknownHostException。所以，我发现我的问题。与ipadress添加奴隶计算机名我的主人的/ etc /主机，然后重新启动的Hadoop的Map Reduce工作。


我们可以检查UI
上述作业日志
1.首先转到jobtracker.jsp

2.单击工作

3.You可以看到目前正在运行什么，或点击地图或减少并检查日志。

Finally I made it. Here is I what I did. I checked the wrong logs when running map reduce job. Each and every job hadoop generate some job logs that locate in logs/userlogs/job_id/* . So when I checking these logs i can finally see some exception that coming know as UnknownHostException. So I found my problem. Add slaves computer name with ipadress to my masters /etc/host and restart the hadoop map reduce job.
We can check the above job logs in UI
1.first goto jobtracker.jsp
2.Click the job
3.You can see what is running right now or click map or reduce and check the logs.

这篇关于Apache的Hadoop的地图reduce任务冰点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！