在mapper的单个输出上运行多个reducer [英] Run multiple reducers on single output from mapper

问题描述

我正在使用map reduce实现左联接功能.左侧大约有6亿条记录，而右侧大约有2300万条记录.在mapper中，我使用左连接条件中使用的列制作键，并将键值输出从mapper传递给reducer. 由于两个表中的映射器键的值都很高(例如分别为456789和78960)，我遇到了性能问题.即使其他减速机完成了工作，这些减速机仍可运行更长的时间. 多个reducer是否可以并行处理来自mapper的相同键值输出以提高性能?

I am implementing a left join functionality using map reduce. Left side is having around 600 million records and right side is having around 23 million records. In mapper I am making the keys using the columns used in left join condition and passing the key-value output from mapper to reducer. I am getting performance issue because of few mapper keys for which number of values in both the tables are high (eg. 456789 and 78960 respectively). Even though other reducers finish their job, these reducers keep running for longer time. Is there any way that multiple reducers can work on the same key-value output from mapper in parallel to better the performance?

这是我要优化的Hive查询.

This is the Hive query that i want to optimize.

select distinct 
        a.sequence, 
        a.fr_nbr, 
        b.to_nbr, 
        a.fr_radius,
        a.fr_zip, 
        a.latitude as fr_latitude, 
        a.longitude as fr_longitude, 
        a.to_zip, 
        b.latitude as to_latitude, 
        b.longitude as to_longitude,
        ((2 * asin( sqrt( cos(radians(a.latitude)) * cos(radians(b.latitude)) * pow(sin(radians((a.longitude - b.longitude)/2)), 2) + pow(sin(radians((a.latitude - b.latitude)/2)), 2) ) )) * 6371 * 0.621371) as distance,
        a.load_year, 
        a.load_month
from common.sb_p1 a LEFT JOIN common.sb__temp0u b    
        on a.to_zip=b.zip
            and a.load_year=b.load_year
            and a.load_month=b.load_month
where   b.correction = 0 
        and a.fr_nbr <> b.to_nbr 
        and ((2 * asin( sqrt( cos(radians(a.latitude)) * cos(radians(b.latitude)) * pow(sin(radians((a.longitude - b.longitude)/2)), 2) + pow(sin(radians((a.latitude - b.latitude)/2)), 2) ) )) * 6371 * 0.621371 <= a.fr_radius)

任何其他解决方案也将不胜感激.

Any other solution will also be appreciated.

推荐答案

使用UNION ALL拆分倾斜的键:

select * from table1 a left join table2 b on a.key=b.key
where a.key not in (456789,78960)
union all
select * from table1 a left join table2 b on a.key=b.key
where a.key = 456789
union all
select * from table1 a left join table2 b on a.key=b.key
where a.key = 78960
;

这些子查询将并行运行，偏斜的密钥将不会分配给单个化简器

These subqueries will run in parallel, skewed keys will not be distributed to single reducer

这篇关于在mapper的单个输出上运行多个reducer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！