Wordcount 程序卡在 hadoop-2.3.0 [英] Wordcount program is stuck in hadoop-2.3.0

问题描述

我安装了 hadoop-2.3.0 并尝试运行 wordcount 示例但它开始工作并闲置

I installed hadoop-2.3.0 and tried to run wordcount example But it starts the job and sits idle

hadoop@ubuntu:~$ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar    wordcount /myprg outputfile1
14/04/30 13:20:40 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/04/30 13:20:51 INFO input.FileInputFormat: Total input paths to process : 1
14/04/30 13:20:53 INFO mapreduce.JobSubmitter: number of splits:1
14/04/30 13:21:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1398885280814_0004
14/04/30 13:21:07 INFO impl.YarnClientImpl: Submitted application application_1398885280814_0004
14/04/30 13:21:09 INFO mapreduce.Job: The url to track the job: http://ubuntu:8088/proxy/application_1398885280814_0004/
14/04/30 13:21:09 INFO mapreduce.Job: Running job: job_1398885280814_0004

跟踪作业的网址:application_1398885280814_0004/

对于以前的版本，我没有遇到这样的问题.我能够在以前的版本中运行 hadoop wordcount.我按照这些步骤安装了hadoop-2.3.0

For previous versions I did nt get such an issue. I was able to run hadoop wordcount in previous version. I followed these steps for installing hadoop-2.3.0

请提出建议.

推荐答案

我在切换到 YARN 时遇到了完全相同的情况.基本上，MRv1 中有 task slot 的概念，MRv2 中有 containers 的概念.这两者在任务在节点上的调度和运行方式上有很大不同.

I had the exact same situation a while back while switching to YARN. Basically there was the concept of task slots in MRv1 and containers in MRv2. Both of these differ very much in how the tasks are scheduled and run on the nodes.

你的工作卡住的原因是它无法找到/启动一个容器.如果您查看 Resource Manager/Application Master 等守护进程的完整日志，您可能会发现它在开始分配新容器后什么也没做.

The reason that your job is stuck is that it is unable to find/start a container. If you go into the full logs of Resource Manager/Application Master etc daemons, you may find that it is doing nothing after it starts to allocate a new container.

要解决这个问题，您必须在yarn-site.xml 和mapred-site.xml 中调整您的内存设置.在我自己做同样的事情时，我发现了this 和 这个 教程特别有用.我建议您尝试使用非常基本的内存设置并稍后对其进行优化.首先检查字数示例，然后再检查其他复杂的示例.

To solve the problem, you have to tweak your memory settings in yarn-site.xml and mapred-site.xml. While doing the same myself, I found this and this tutorials especially helpful. I would suggest you to try with the very basic memory settings and optimize them later on. First check with a word count example then go on to other complex ones.

这篇关于Wordcount 程序卡在 hadoop-2.3.0的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！