如何访问输出阶段的 Mapper/Reducer 计数器? [英] How can I access the Mapper/Reducer counters on the Output stage?

问题描述

我在 Mapper 课程中创建了一些计数器:

I have some counters I created at my Mapper class:

(使用 appengine-mapreduce Java 库 v.0.5 编写的示例)

(example written using the appengine-mapreduce Java library v.0.5)

@Override
public void map(Entity entity) {
    getContext().incrementCounter("analyzed");
    if (isSpecial(entity)){
        getContext().incrementCounter("special");
    }
}

(方法 isSpecial 只是根据实体的状态返回 true 或 false，与问题无关)

(The method isSpecial just returns true or false depending on the state of the entity, not relevant to the question)

我想在处理完所有内容后访问这些计数器，在 Output 类的 finish 方法中:

I want to access those counters when I finish processing the whole stuff, at the finish method of the Output class:

@Override
public Summary finish(Collection<? extends OutputWriter<Entity>> writers) {
    //get the counters and save/return the summary
    int analyzed = 0; //getCounter("analyzed");
    int special = 0; //getCounter("special");
    Summary summary = new Summary(analyzed, special);
    save(summary);
    return summary;
}

...但是 getCounter 方法只能从 MapperContext 类，只能通过 Mappers/Reducers getContext() 方法访问.

... but the method getCounter is only available from the MapperContext class, which is accessible only from Mappers/Reducers getContext() method.

如何在输出阶段访问我的计数器?

How can I access my counters at the Output stage?

旁注:我无法将计数器值发送到我的输出类，因为整个 Map/Reduce 是将一组实体转换为另一组(换句话说:计数器不是 Map/减少).计数器仅用于控制 - 我在这里计算它们而不是创建另一个进程只是为了进行计数是有道理的.

Side note: I can't send the counters values to my outputted class because the whole Map/Reduce is about transforming a set of Entities to another set (in other words: the counters are not the main purpose of the Map/Reduce). The counters are just for control - it makes sense I compute them here instead of creating another process just to make the counts.

谢谢.

推荐答案

今天没有办法在输出中做到这一点.但请随时在此处请求:https://code.google.com/p/appengine-mapreduce/issues/list

There is not a way to do this inside of output today. But feel free to request it here: https://code.google.com/p/appengine-mapreduce/issues/list

但是，您可以做的是链接一个作业以在您的 map-reduce 之后运行，该作业将接收它的输出和计数器.这里有一个例子:https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/example/src/com/google/appengine/demos/mapreduce/entitycount/ChainedMapReduceJob.java

What you can do however is to chain a job to run after your map-reduce that will receive it's output and counters. There is an example of this here: https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/example/src/com/google/appengine/demos/mapreduce/entitycount/ChainedMapReduceJob.java

在上面的示例中，它连续运行 3 个 MapReduce 作业.请注意，这些不一定是 MapReduce 作业，您可以创建自己的类来扩展 Job 并具有创建 Summary 对象的 run 方法.

In the above example it is running 3 MapReduce jobs in a row. Note that these don't have to be MapReduce jobs, you can create your own class that extends Job and has a run method which creates your Summary object.

这篇关于如何访问输出阶段的 Mapper/Reducer 计数器?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！