Hadoop中的默认哈希分区程序将哪两个不同的键分配给相同的reducer? [英] What two different keys go to the same reducer by the default hash partitioner in Hadoop?

问题描述

我们知道Hadoop保证来自不同mappers的相同密钥将被发送到相同的reducer.

As we know that Hadoop guarantees that the same keys which come from different mappers will be sent to the same reducer.

但是，如果两个不同的键具有相同的哈希值，则它们肯定会进入相同的 reducer，因此它们将被发送到相同的 reduce方法有序?哪一部分负责此逻辑?

But if two different keys have the same hash value, they definitely will go to the same reducer, so will them be sent to the same reduce method orderly? Which part is responsible for this logic?

非常感谢！

推荐答案

Q1:so will them be sent to the same reduce method orderly

Ans:是

第二季度:Which part is responsible for this logic

答案:随机排序

示例:

key  value
 1       2
 1       2
 2       5
 3       19
 6       20

假设减速器的数量为5 所以现在.

Lets say number of reducer is 5 so now .

Reduce 0 will get key NO key-value pairs
Reduce 1 will get key 1,6 in same order 
Reduce 2 will get key 2
Reduce 3 will get key 3
Reduce 4 will get key NO key-value pairs

这篇关于Hadoop中的默认哈希分区程序将哪两个不同的键分配给相同的reducer?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！