从减速器访问映射器的计数器 [英] Accessing a mapper's counter from a reducer

问题描述

我需要从我的减速器中的映射器访问计数器。这可能吗？如果是这样怎么办？

举个例子：
我的映射器是：

  public class CounterMapper扩展Mapper<文本，文本，文本，文本> {
 
静态枚举TestCounters {TEST} 
 
 @Override $ b $保护无效映射（文本键，文本值，上下文上下文）
抛出IOException，InterruptedException {
 context.getCounter（TestCounters.TEST）.increment（1）; 
 context.write（key，value）; 
} 
} 
  

我的减速器是

  public class CounterReducer extends Reducer< Text，Text，Text，LongWritable> 
 $ b @Override 
 protected void reduce（Text key，Iterable< Text> values，Context context）
 throws IOException，InterruptedException {
 Counter counter = context.getCounter （CounterMapper.TestCounters.TEST）; 
 long counterValue = counter.getValue（）; 
 context.write（key，new LongWritable（counterValue））; 
 
 
 $ / code $ / pre 
 $ b $ p counterValue总是0. 
我是I做错了什么或者这是不可能的？
解决方案
在Reducer的配置（JobConf）中，可以使用JobConf对象查看减速器自己的工作标识。因此，您的Reducer可以创建自己的JobClient  - 即与jobtracker的连接 - 并查询计数器以查找此作业（或针对此问题的任何作业）。 
 
 
  //在Reducer类中... 
 private long mapperCounter; 
 
 @Override 
 public void configure（JobConf conf）{
 JobClient client = new JobClient（conf）; 
 RunningJob parentJob = 
 client.getJob（JobID.forName（conf.get（mapred.job.id）））; 
 mapperCounter = parentJob.getCounters（）。getCounter（MAP_COUNTER_NAME）; 
 
 
 
 $ b现在你可以在reduce（）方法本身内使用mapperCounter。
 
 
 
您实际上需要尝试在这里尝试。我使用旧的API，但它不应该很难适应新的API。
 
 
 
请注意，mappers的计数器应该在任何reducer启动之前完成，所以与贾斯廷托马斯的评论相反，我相信你应该得到准确的价值（只要减价者不增加同一个计数器！）
I need to access the counters from my mapper in my reducer.  Is this possible?  If so how is it done?

As an example:
my mapper is:
public class CounterMapper extends Mapper<Text,Text,Text,Text> {

    static enum TestCounters { TEST }

    @Override
    protected void map(Text key, Text value, Context context)
                    throws IOException, InterruptedException {
        context.getCounter(TestCounters.TEST).increment(1);
        context.write(key, value);
    }
}
My reducer is
public class CounterReducer extends Reducer<Text,Text,Text,LongWritable> {

    @Override
    protected void reduce(Text key, Iterable<Text> values, Context context)
                        throws IOException, InterruptedException {
        Counter counter = context.getCounter(CounterMapper.TestCounters.TEST);
        long counterValue = counter.getValue();
        context.write(key, new LongWritable(counterValue));
    }
}
counterValue is always 0.
Am I doing something wrong or is this just not possible?
解决方案
In the Reducer's configure(JobConf), you can use the JobConf object to look up the reducer's own job id.  With that, your reducer can create its own JobClient -- i.e. a connection to the jobtracker -- and query the counters for this job (or any job for that matter).
// in the Reducer class...
private long mapperCounter;

@Override
public void configure(JobConf conf) {
    JobClient client = new JobClient(conf);
    RunningJob parentJob = 
        client.getJob(JobID.forName( conf.get("mapred.job.id") ));
    mapperCounter = parentJob.getCounters().getCounter(MAP_COUNTER_NAME);
}
Now you can use mapperCounter inside the reduce() method itself.

You actually need a try-catch here.  I'm using the old API, but it shouldn't be hard to adapt for the new API.

Note that mappers' counters should all be finalized before any reducer starts, so contrary to Justin Thomas's comment, I believe you should get accurate values (as long as the reducers aren't incrementing the same counter!)

                        
这篇关于从减速器访问映射器的计数器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！