使用Hadoop MapRed排序 [英] Sort order with Hadoop MapRed

问题描述

那么，

我想知道如何在reduce任务后更改简单的WordCount程序的排序顺序？我已经制作了另一张地图，通过按键进行价值排序，但仍然按升序排列。
有没有简单的方法来做到这一点（改变排序顺序）？！

感谢
Vellozo

如果你使用的是旧的API（ mapred。* ），那么在作业中设置OutputKeyComparatorClass conf：

  jobConf.setOutputKeyComparatorClass（ReverseComparator.class）; 
  

ReverseComparator可以是这样的：

  static class ReverseComparator extends WritableComparator {
 private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator（）; 
 
 public ReverseComparator（）{
 super（Text.class）; 
} 
 
 @Override 
 public int compare（byte [] b1，int s1，int l1，byte [] b2，int s2，int l2）{
尝试{
 return（-1）* TEXT_COMPARATOR 
 .compare（b1，s1，l1，b2，s2，l2）; 
} catch（IOException e）{
 throw new IllegalArgumentException（e）; 
 
 
 $ b $ @Override 
 public int compare（WritableComparable a，WritableComparable b）{
 if（Text&& b instanceof文本）{
 return（-1）*（（（Text）a）
 .compareTo（（Text）b）））; 
} 
返回super.compare（a，b）; 
 
 
 
 
 
 
在新API（ mapreduce。* ），我想你需要使用 Job.setSortComparator（）方法。
 
Well,

I'd like to know how can I change the sort order of my simple WordCount program after the reduce task? I've already made another map to order by value instead by keys, but it still ordered in ascending order.
Is there an easy simple method to do this (change the sort order)?!

Thanks
Vellozo
解决方案
If you are using the older API (mapred.*), then set the OutputKeyComparatorClass in the job conf: 
jobConf.setOutputKeyComparatorClass(ReverseComparator.class);
ReverseComparator can be something like this:
static class ReverseComparator extends WritableComparator {
        private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator();

        public ReverseComparator() {
            super(Text.class);
        }

        @Override
        public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
            try {
                return (-1)* TEXT_COMPARATOR
                        .compare(b1, s1, l1, b2, s2, l2);
            } catch (IOException e) {
                throw new IllegalArgumentException(e);
            }
        }

        @Override
        public int compare(WritableComparable a, WritableComparable b) {
            if (a instanceof Text && b instanceof Text) {
                return (-1)*(((Text) a)
                        .compareTo((Text) b)));
            }
            return super.compare(a, b);
        }
    }
In the new API (mapreduce.*), I think you need to use the Job.setSortComparator() method.

                        
这篇关于使用Hadoop MapRed排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！