hadoop集群中的Mapreduce执行 [英] Mapreduce execution in a hadoop cluster
问题描述
我对 Mapreduce的工作原理 有点困惑.我读了一些文章，但没有得到正确的答案.
I am a bit confused about How exactly the Mapreduce works. I have read some articles but didn't get the proper answer.
场景:
我在HDFS上存储了一个1 TB大小的文件(假设它存储在/user/input/位置).复制为3，块大小为128 MB.
I stored a file of size 1 TB on top of HDFS (Let's say it is stored at a location /user/input/ ). Replication is 3 and the block size 128 MB.
现在，我想使用mapreduce分析此1TB文件.由于块大小为128 MB，所以我总共将有8192个块.考虑到我集群中有100台机器，
Now, I want to analyze this 1TB file using mapreduce. Since the block size is 128 MB, I will have 8192 blocks in total.Considering I have 100 machines in the cluster then
将在所有100个节点上产生8192个映射任务，并平均分配映射器的数量吗?还是仅在放置复制数据的那些节点上运行?
Will 8192 map tasks will spawned on all the 100 nodes, evenly distributing the number of mappers ? Or it will run on only those nodes where replicated data is placed ?
推荐答案
映射器的数量取决于 InputSplits ，而不取决于复制因子.
Number of Mappers depend on InputSplits and not on replication factor.
有关理解InputSplits内部的信息，请参见以下帖子:
Refer to below post for understanding internals of InputSplits:
映射器和缩减器的数量由Hadoop框架决定.
The number of mappers and reducers are decided by Hadoop Framework.
有关更多详细信息，请参见以下帖子:
Refer to below post for more details:
为简单起见，假定HDFS块和InputSplit相同，而数据不跨越多个数据节点.
For simplicity sake, assume that HDFS block and InputSplit are same without data spanning across multiple data nodes.
对于您来说， 1 TB 文件处理需要 8192 地图.启动Map任务时，map任务会尝试在存在数据的节点上运行Mapper. 8192 个块的1 TB文件可能未在 100 个节点上平均分配.如果它们均匀分布在 100 个节点上，则Framework将在所有 100 个节点上运行地图任务. 数据局部性在选择数据节点中起着关键作用.
In your case, 1 TB file processing requires 8192 Maps. While starting Map task, map task tries to run Mapper on the node where data is present. 1 TB file of 8192 blocks may not have evenly distributed on 100 nodes. If they are evenly distributed on 100 nodes, Framework will run the map tasks on all 100 nodes. Data locality plays key role in selection of data node.
这篇关于hadoop集群中的Mapreduce执行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
