MapReduce 按值降序排序 [英] MapReduce sort by value in descending order

问题描述

我正在尝试用伪代码编写一个 MapReduce 任务，该任务返回按降序排序的项目.例如:对于 wordcount 任务，而不是获取:

I'm trying to write in pseudo code a MapReduce task that returns the items sorted in descending order. For example: for the wordcount task, instead of getting:

apple 1
banana 3
mango 2

我希望输出是:

banana 3
mango 2
apple 1

关于如何做的任何想法?我知道如何按升序(替换映射器作业中的键和值)而不是降序.

Any ideas of how to do it? I know how to do it in ascending order (replace the keys and value in the mapper job) but not in descending order.

推荐答案

这里可以借助下面的reducer代码实现降序排序.

Here you can take help of below reducer code to achieve sorting in descending order .

假设您已经编写了映射器和驱动程序代码，其中映射器将生成输出为 (Banana,1) 等

Assuming you have written mapper and driver code where mapper will produce output as (Banana,1) etc

在 reducer 中，我们将对特定键的所有值求和，并将最终结果放入 map 中，然后根据值对 map 进行排序，并将最终结果写入 reduce 的清理函数中.

In reducer we will sum all values for a particular key and put final result in a map then sort the map on the basis of values and write final result in cleanup function of reduce.

请参阅下面的代码以了解更多信息:

Please see below code for further understadnind:

public class Word_Reducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    // Change access modifier as per your need 
    public Map<String , Integer > map = new LinkedHashMap<String , Integer>();
    public void reduce(Text key , Iterable<IntWritable> values ,Context context)
    { 
    // write logic for your reducer 
    // Enter reduced values in map for each key
    for (IntWritable value : values ){
         // calculate "count" associated with each word 
    }
    map.put(key.toString() , count); 
}

public void cleanup(Context context){ 
    //Cleanup is called once at the end to finish off anything for reducer
    //Here we will write our final output
    Map<String , Integer>  sortedMap = new HashMap<String , Integer>();    
    sortedMap = sortMap(map);

    for (Map.Entry<String,Integer> entry = sortedMap.entrySet()){
        context.write(new Text(entry.getKey()),new IntWritable(entry.getValue()));
    }
}

public Map<String , Integer > sortMap (Map<String,Integer> unsortMap){
    Map<String ,Integer> hashmap = new LinkedHashMap<String,Integer>();
    int count=0;
    List<Map.Entry<String,Integer>> list = new 
    LinkedList<Map.Entry<String,Integer>>(unsortMap.entrySet());
    //Sorting the list we created from unsorted Map
    Collections.sort(list , new Comparator<Map.Entry<String,Integer>>(){
        public int compare (Map.Entry<String , Integer> o1 , Map.Entry<String , Integer> o2 ){
            //sorting in descending order
            return o2.getValue().compareTo(o1.getValue());
        }
    });

    for(Map.Entry<String, Integer> entry : list){
        // only writing top 3 in the sorted map 
        if(count>2)
            break;
        hashmap.put(entry.getKey(),entry.getValue());
    }
    return hashmap ; 
}

这篇关于MapReduce 按值降序排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！