Hadoop中的映射器输出保存在哪里？ [英] Where Mapper output in Hadoop is saved?

问题描述

我有兴趣有效地管理Hadoop洗牌流量并有效利用网络带宽。要做到这一点，我想知道每个Datanodes生成的流量是多少？对流量进行混洗只不过是映射器的输出。所以这个映射器输出保存在哪里？我怎样才能实时从每个datanodes获得mapper输出的大小？感谢您的帮助。

我已创建一个目录来存储此映射器输出，如下所示。

 <属性> 
<名称> mapred.local.dir< / name> 
< value> / app / hadoop / tmp / myoutput< / value> 
< / property> 
  

我看了

  hduser @ dn4：/ app / hadoop / tmp / myoutput $ ls -lrt 
 total 16 
 drwxr-xr-x 2 hduser hadoop 4096 Dec 12 10:50 tt_log_tmp 
 drwx ------ 3 hduser hadoop 4096 Dec 12 10:53 ttprivate 
 drwxr-xr-x 3 hduser hadoop 4096 Dec 12 10:53 taskTracker 
 drwxr-xr-x 4 hduser hadoop 4096 Dec 12 13:25 userlogs 
  

当我运行mapreduce作业时，我无法在这里找到任何东西。 / p>

谢谢

解决方案

地图作业的输出存储在映射器的本地磁盘。一旦地图作业完成，这些本地输出就会转移到缩减器。您可以检查您的 $ HADOOP_HOME / conf / mapred-site.xml 以检查映射器输出的存储位置。

 <性> 
<名称> mapred.local.dir< / name> 
<值> $ DIR< /值> 
< / property> 
  

I am interested in efficiently manage the Hadoop shuffling traffic and utilize the network bandwidth effectively. To do this I want to know how much shuffling traffic generated by each Datanodes ? Shuffling traffic is nothing but the output of mappers. So where this mapper output is saved ? How can i get the size of mapper output from each datanodes in a real time ? Appreciate your help.

I have created a directory to store this mapper output as below.

 <property>
 <name>mapred.local.dir</name>
 <value>/app/hadoop/tmp/myoutput</value>
 </property>                     

and i looked at

 hduser@dn4:/app/hadoop/tmp/myoutput$ ls -lrt
 total 16
 drwxr-xr-x 2 hduser hadoop 4096 Dec 12 10:50 tt_log_tmp
 drwx------ 3 hduser hadoop 4096 Dec 12 10:53 ttprivate
 drwxr-xr-x 3 hduser hadoop 4096 Dec 12 10:53 taskTracker
 drwxr-xr-x 4 hduser hadoop 4096 Dec 12 13:25 userlogs  

and i couldnot find anything here when i run the mapreduce job .

Thanks

解决方案

The output of the maps jobs is stored in the local disk of the mappers. Once the map job finishes these local outputs are then transferred to reducers. You can check your $HADOOP_HOME/conf/mapred-site.xml to check where mapper outputs are stored.

<property>
    <name>mapred.local.dir</name>
    <value>$DIR</value>
</property>

这篇关于Hadoop中的映射器输出保存在哪里？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！