mapper和reducer函数的输出是什么 [英] What exactly is output of mapper and reducer function

问题描述

这是使用mapReduce和hadoop提取包含特定值的行的后续问题


Mapper函数

  public static class MapForWordCount扩展Mapper< Object，Text，Text，IntWritable> {
 
 private IntWritable saleValue = new IntWritable（）; 
私人文本rangeValue = new Text（）; 
 $ b $ public void map（Object key，Text value，Context con）throws IOException，InterruptedException 
 {
 String line = value.toString（）; 
 String [] words = line.split（，）; 
 for（String word：words）
 {
 if（words [3] .equals（40））{
 saleValue.set（Integer.parseInt（words [0 ]））; 
 rangeValue.set（words [3]）; 
 con.write（rangeValue，saleValue）; 
} 
} 
} 
} 
  

< Reducer函数

  public static class ReduceForWordCount extends Reducer< Text，IntWritable，Text，IntWritable> 
 {
 private IntWritable result = new IntWritable（）; 
 public void reduce（Text word，Iterable< IntWritable> values，Context con）throws IOException，InterruptedException 
 {
 for（IntWritable value：values）
 {
 result .SET（value.get（））; 
 con.write（word，result）; 
} 
} 
} 
  

  40 105 
 40 105 
 40 105 
 40 105 
  

编辑1：
但期望的输出是

  40 102 
 40 104 
 40 105 
  

我在做什么错了？

在mapper和reducer中究竟发生了什么函数？

解决方案

在原始问题 - 你不需要循环不在映射器或reducer中，因为你正在复制条目：

pre $ public static class MapForWordCount扩展Mapper< Object，Text，Text，IntWritable> {
$ b $ private intWritable saleValue = new IntWritable （）;
私人文本rangeValue = new Text（）;
$ b $ public void map（Object key，Text value，Context con）throws IOException，InterruptedException
{
String line = value.toString（）;
String [] words = line.split（，）;
if（words [3] .equals（40））{
saleValue.set（Integer.parseInt（words [0]））;
rangeValue.set（words [3]）;
con.write（rangeValue，saleValue）;
}
}
}


而在减速器中，as @Serhiy在原始问题中建议您只需要一行代码：

  public static class ReduceForWordCount extends Reducer< Text，IntWritable ，文字，IntWritable> 
 {
 private IntWritable result = new IntWritable（）; 
 public void reduce（Text text，Iterable< IntWritable> values，Context con）throws IOException，InterruptedException 
 {
 con.write（word，null）; 
} 
  

重写编辑1 - 我会留下一个微不足道的练习：）

This is a follow up question of Extracting rows containing specific value using mapReduce and hadoop
Mapper function

public static class MapForWordCount extends Mapper<Object, Text, Text, IntWritable>{

private IntWritable saleValue = new IntWritable();
private Text rangeValue = new Text();

public void map(Object key, Text value, Context con) throws IOException, InterruptedException
{
    String line = value.toString();
    String[] words = line.split(",");
    for(String word: words )
    {
        if(words[3].equals("40")){  
            saleValue.set(Integer.parseInt(words[0]));
            rangeValue.set(words[3]);
            con.write( rangeValue , saleValue );
        }
    }
}   
}

Reducer function

public static class ReduceForWordCount extends Reducer<Text, IntWritable, Text, IntWritable>  
{  
    private IntWritable result = new IntWritable();  
    public void reduce(Text word, Iterable<IntWritable> values, Context con) throws IOException, InterruptedException  
    {  
        for(IntWritable value : values)  
        {  
            result.set(value.get());  
            con.write(word, result);  
        }  
    }  
}

Output obtained is

40 105  
40 105  
40 105  
40 105

EDIT 1 : But the Expected output is

40 102  
40 104  
40 105

What am I doing wrong ?

What exactly is happening here in mapper and reducer function ?

解决方案

In the context of the original question - you don't need the loop not in the mapper nor in the reducer as you are duplicating entries:

public static class MapForWordCount extends Mapper<Object, Text, Text, IntWritable>{

private IntWritable saleValue = new IntWritable();
private Text rangeValue = new Text();

public void map(Object key, Text value, Context con) throws IOException, InterruptedException
{
    String line = value.toString();
    String[] words = line.split(",");
    if(words[3].equals("40")){  
       saleValue.set(Integer.parseInt(words[0]));
       rangeValue.set(words[3]);
       con.write(rangeValue , saleValue );
    }
}   
}

And in the reducer, as suggested by @Serhiy in the original question you need only one line of code:

public static class ReduceForWordCount extends Reducer<Text, IntWritable, Text, IntWritable>  
{  
private IntWritable result = new IntWritable();  
public void reduce(Text word, Iterable<IntWritable> values, Context con) throws IOException, InterruptedException  
{  
    con.write(word, null);  
} 

Regrading "Edit 1" - I will leave it a trivial practice :)

这篇关于mapper和reducer函数的输出是什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！