Hadoop 2.2.0中的MapReduce无法正常工作 [英] MapReduce in Hadoop 2.2.0 not working

问题描述

在伪分布式模式下安装并配置我的 Hadoop 2.2.0 后，所有内容都正在运行，您可以在 jps
$ b $ $ $ $
$ $ $ $ $ $ $ $ $ 2287 JobHistoryServer
1926 ResourceManager
2162 NodeManager
1834 DataNode
1756 NameNode
3013 Jps

然后我运行wordcount示例

  hadoop jar $ HADOOP_HOME / share / hadoop / mapreduce / hadoop-mapreduce-examples-2.2.0.jar wordcount / user / hdfs / file / output 
  

执行frezees（？）如下：

  $ hadoop jar $ HADOOP_HOME / share / hadoop / mapreduce / hadoop-mapreduce-examples-2.2.0.jar wordcount / user / hdfs /文件/输出
 OpenJDK 64位服务器VM警告：您已经加载了库/home/hduser/hadoop-src/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0，这可能会禁用堆栈守卫。虚拟机现在会尝试修复堆栈后卫。 
强烈建议您使用'execstack -c< libfile>'来修复库，或者将它与'-z noexecstack'链接起来。 
 14/04/22 22:17:24 WARN util.NativeCodeLoader：无法为您的平台加载native-hadoop库......在适用的情况下使用builtin-java类
 14/04/22 22： 17:25 INFO client.RMProxy：连接到ResourceManager，位于/192.168.33.10:8032 
 14/04/22 22:17:25 INFO input.FileInputFormat：要处理的总输入路径：1 
 14 / 04/22 22:17:25信息mapreduce.JobSubmitter：拆分数量：1 
 14/04/22 22:17:25信息Configuration.deprecation：user.name已弃用。相反，使用mapreduce.job.user.name 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.jar已弃用。相反，使用mapreduce.job.jar 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.output.value.class已弃用。相反，使用mapreduce.job.output.value.class 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapreduce.combine.class已弃用。相反，使用mapreduce.job.combine.class 
 14/04/22 22:17:25信息Configuration.deprecation：mapreduce.map.class已弃用。相反，使用mapreduce.job.map.class 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.job.name已弃用。相反，使用mapreduce.job.name 
 14/04/22 22:17:25信息Configuration.deprecation：mapreduce.reduce.class已弃用。相反，使用mapreduce.job.reduce.class 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.input.dir已弃用。相反，使用mapreduce.input.fileinputformat.inputdir 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.output.dir已弃用。相反，使用mapreduce.output.fileoutputformat.outputdir 
 14/04/22 22:17:25信息Configuration.deprecation：mapred.map.tasks已弃用。相反，使用mapreduce.job.maps 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.output.key.class已弃用。相反，使用mapreduce.job.output.key.class 
 14/04/22 22:17:25 INFO Configuration.deprecation：mapred.working.dir已弃用。相反，使用mapreduce.job.working.dir 
 14/04/22 22时17分26秒INFO mapreduce.JobSubmitter：提交令牌任务：job_1398204897594_0002 
 14/04/22 22时17分26秒INFO impl.YarnClientImpl：HTTP：跟踪工作的网址：在/192.168.33.10:8032 
 14/04/22 22时十七分26秒INFO mapreduce.Job提交申请application_1398204897594_0002到ResourceManager的无业游民//  -  ubuntu-俏皮-64：8088 /代理/ application_1398204897594_0002 / 
 14/04/22 22时十七分26秒INFO mapreduce.Job：正在运行的作业：job_1398204897594_0002 
 14/04/22 22时十七分36秒INFO mapreduce的。 Job：job job_1398204897594_0002以超级模式运行：false 
 14/04/22 22:17:36信息mapreduce.Job：map 0％reduce 0％
  

有什么想法？

解决方案

问题出在文件纱-site.xml中。 属性必须大于 3072 Mb ，并且我将它配置为 1024 Mb ，所以正确的方法是

 < property> 
< name> yarn.nodemanager.resource.memory-mb< / name> 
<值> 4096< /值> 
< description>物理内存（以MB为单位）可用于运行容器< / description> 
< / property> 
  

After installing and configuring my Hadoop 2.2.0 in pseudo-distributed mode everything is running, as you can see in the jps:

$ jps
2287 JobHistoryServer
1926 ResourceManager
2162 NodeManager
1834 DataNode
1756 NameNode
3013 Jps

Then I ran the wordcount example with

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /user/hdfs/file /output

And the execution frezees (?) as follows:

$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /user/hdfs/file /output                                          
OpenJDK 64-Bit Server VM warning: You have loaded library /home/hduser/hadoop-src/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/04/22 22:17:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/22 22:17:25 INFO client.RMProxy: Connecting to ResourceManager at /192.168.33.10:8032
14/04/22 22:17:25 INFO input.FileInputFormat: Total input paths to process : 1
14/04/22 22:17:25 INFO mapreduce.JobSubmitter: number of splits:1
14/04/22 22:17:25 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/04/22 22:17:25 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/04/22 22:17:25 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/04/22 22:17:25 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/04/22 22:17:25 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/04/22 22:17:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1398204897594_0002
14/04/22 22:17:26 INFO impl.YarnClientImpl: Submitted application application_1398204897594_0002 to ResourceManager at /192.168.33.10:8032
14/04/22 22:17:26 INFO mapreduce.Job: The url to track the job: http://vagrant-ubuntu-saucy-64:8088/proxy/application_1398204897594_0002/
14/04/22 22:17:26 INFO mapreduce.Job: Running job: job_1398204897594_0002
14/04/22 22:17:36 INFO mapreduce.Job: Job job_1398204897594_0002 running in uber mode : false
14/04/22 22:17:36 INFO mapreduce.Job:  map 0% reduce 0%

Any ideas?

解决方案

The problem was in the file yarn-site.xml. The property must be larger than 3072 Mb, and I had it configured to 1024 Mb, so the correct way is

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>4096</value>
    <description>Physical memory, in MB, to be made available to running containers</description>
</property>

这篇关于Hadoop 2.2.0中的MapReduce无法正常工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！