MapReduce按值按降序排序 [英] MapReduce sort by value in descending order
问题描述
我尝试在伪代码中写入一个MapReduce任务，该任务返回按降序排序的项目。例如：为wordcount任务，而不是得到：
apple 1
香蕉3
芒果2
我希望输出为：
香蕉3
芒果2
苹果1
有关如何做的任何想法？我知道如何按照升序来做（取代mapper作业中的键和值），但不是按降序排列。
>假设你已经编写了映射器和驱动程序代码，其中映射器将生成输出为（Banana，1 ）等。
在缩减器中，我们将总和一个特定键的所有值并将最终结果放入一个映射中，然后根据值对映射进行排序，并在清理中写入最终结果减少功能。
请参阅下面的代码以获取更多信息：
public class Word_Reducer extends Reducer< Text，IntWritable，Text，
IntWritable> {
//根据需要更改访问修饰符
public Map< String，Integer> map = new LinkedHashMap< String，Integer>（）;
public void reduce（Text key，Iterable< IntWritable> values，Context context
）
{
//为您的reducer写入逻辑
//为每个键在映射中输入缩减的值
（IntWritable value：values）{
//计算与每个单词关联的count
}
map.put（key.toString（），count）;
}
public void cleanup（Context context）{
//清理在最后被调用一次以完成任何事情for reducer
//这里我们将写出我们的最终输出
Map< String，Integer> sortedMap = new HashMap< String，Integer>（）;
/
sortedMap = sortMap（map）; （Map.Entry< String，Integer> entry = sortedMap.entrySet（））{
context.write（new Text（entry.getKey（）），new
（
） IntWritable（entry.getValue（）））;
}
}
public Map< String，Integer> sortMap（Map< String，Integer> unsortMap）{
Map< String，Integer> hashmap = new LinkedHashMap< String，Integer>（）;
int count = 0;
列表< Map.Entry< String，Integer>> list = new
LinkedList< Map.Entry< String，Integer>>（unsortMap.entrySet（））;
//对从未排序的Map创建的列表进行排序Map
Collections.sort（list，new Comparator< Map.Entry< String，Integer>>（）{
public int比较（Map.Entry< String，Integer> o1，Map.Entry< String，
Integer> o2）{
//按降序排序
返回o2.getValue（）。compareTo o1.getValue（））;
}
}）; （Map.Entry< String，Integer> entry：list）{
//只在有序地图中写入前3个
if（count> 2）$ b $
break;
hashmap.put（entry.getKey（），entry.getValue（））;
}
返回hashmap;
}
}
I'm trying to write in pseudo code a MapReduce task that returns the items sorted in descending order. For example: for the wordcount task, instead of getting:
apple 1
banana 3
mango 2
I want the output to be:
banana 3
mango 2
apple 1
Any ideas of how to do it? I know how to do it in ascending order (replace the keys and value in the mapper job) but not in descending order.
Here you can take help of below reducer code to achieve sorting in descending order .
Assuming you have written mapper and driver code where mapper will produce output as (Banana,1) etc
In reducer we will sum all values for a particular key and put final result in a map then sort the map on the basis of values and write final result in cleanup function of reduce.
Please see below code for further understadnind:
public class Word_Reducer extends Reducer<Text,IntWritable, Text ,
IntWritable> {
// Change access modifier as per your need
public Map<String , Integer > map = new LinkedHashMap<String , Integer>();
public void reduce(Text key , Iterable<IntWritable> values ,Context context
)
{
// write logic for your reducer
// Enter reduced values in map for each key
for (IntWritable value : values ){
// calculate "count" associated with each word
}
map.put(key.toString() , count);
}
public void cleanup(Context context){
//Cleanup is called once at the end to finish off anything for reducer
//Here we will write our final output
Map<String , Integer> sortedMap = new HashMap<String , Integer>();
/
sortedMap = sortMap(map);
for (Map.Entry<String,Integer> entry = sortedMap.entrySet()){
context.write(new Text(entry.getKey()),new
IntWritable(entry.getValue()));
}
}
public Map<String , Integer > sortMap (Map<String,Integer> unsortMap){
Map<String ,Integer> hashmap = new LinkedHashMap<String,Integer>();
int count=0;
List<Map.Entry<String,Integer>> list = new
LinkedList<Map.Entry<String,Integer>>(unsortMap.entrySet());
//Sorting the list we created from unsorted Map
Collections.sort(list , new Comparator<Map.Entry<String,Integer>>(){
public int compare (Map.Entry<String , Integer> o1 , Map.Entry<String ,
Integer> o2 ){
//sorting in descending order
return o2.getValue().compareTo(o1.getValue());
}
});
for(Map.Entry<String, Integer> entry : list){
// only writing top 3 in the sorted map
if(count>2)
break;
hashmap.put(entry.getKey(),entry.getValue());
}
return hashmap ;
}
}
这篇关于MapReduce按值按降序排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
