无法在hadoop 2.4.0上运行MapReduce作业 [英] Can't run a MapReduce job on hadoop 2.4.0
问题描述
我是hadoop的新手,这是我的问题。我在3台机器的簇上配置了hadoop 2.4.0和jdk1.7.60。我能够执行hadoop的所有命令。现在我修改了wordcount示例并创建了jar文件。我已经在hadoop 1.2.1上执行了这个jar文件并得到了结果。但现在在hadoop 2.4.0上我没有收到任何结果。
I am new to hadoop and here is my problem. I have configured hadoop 2.4.0 with jdk1.7.60 on cluster of 3 machine. I am able to execute all the commands of hadoop. Now I have modified wordcount example and created jar file. I have already executed with this jar file on hadoop 1.2.1 and got the result. But now on hadoop 2.4.0 I am not getting any result.
用于执行的命令
Command used for execution
$hadoop jar WordCount.jar WordCount /data/webdocs.dat /output
我收到以下消息:
I am getting following message from the setup:
14/06/29 19:35:18 INFO client.RMProxy: Connecting to ResourceManager at /192.168.2.140:8040
14/06/29 19:35:18 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/06/29 19:35:19 INFO input.FileInputFormat: Total input paths to process : 1
14/06/29 19:35:19 INFO mapreduce.JobSubmitter: number of splits:12
14/06/29 19:35:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1403905542893_0004
14/06/29 19:35:19 INFO impl.YarnClientImpl: Submitted application application_1403905542893_0004
14/06/29 19:35:19 INFO mapreduce.Job: The url to track the job: http://192.168.2.140:8088/proxy/application_1403905542893_0004/
14/06/29 19:35:19 INFO mapreduce.Job: Running job: job_1403905542893_0004
此时没有任何信息更改。我等了15到20分钟,但仍然是相同的。
At this point no message change. I waited for 15 to 20 minutes but still the same.
这是我在资源经理的网页上看到的有关工作的内容:
This is what I see on resource manager's web page regarding the job:
State - ACCEPTED
FinalStatus - UNDEFINED
Progress - (progress bar in 0%)
Tracking UI - UNASSIGNED
Apps Submitted - 1
Apps Pending - 1
Apps Running - 0
I tried the other yarn command for execution but got the same result
$yarn jar WordCount.jar WordCount /data/webdocs.dat /output
以下是jps的输出:
Here is the output of jps:
21485 NameNode
23142 DataNode
28504 Jps
21704 ResourceManager
22082 JobHistoryServer
任何帮助或指导都将得到高度评价。
Any help or guidance will be highly appriciated.
推荐答案
我解决了这个问题。这是hadoop配置文件中的错误。
资源管理器端口8040上存在绑定异常。
I solved the problem. It was the mistake in the configuration file of the hadoop. There was bind exception on the port 8040 for resourcemanager.
我将hadoop yarn-site.xml从(旧yarn-site.xml)更改为
I changed the hadoop yarn-site.xml from (old yarn-site.xml):
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.2.140:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.2.140:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.2.140:8040</value>
</property>
</configuration>
到(new yarn-site.xml):
To (new yarn-site.xml):
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
我删除了hadoop配置中的其他行
然后我使用以下命令启动resourcemanager和nodemanager
I deleted the other line in hadoop configuration Then I gace following commands to start resourcemanager and nodemanager
$yarn-daemon.sh start nodemanager
$yarn-daemon.sh start resourcemanager
然后我尝试执行我的工作并取得成功。
Then I tried executing my job and it was successfull.
这篇关于无法在hadoop 2.4.0上运行MapReduce作业的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!