Hadoop:在 MapReduce 期间 OutputCollector 如何工作? [英] Hadoop: How does OutputCollector work during MapReduce?

问题描述

我想知道 Map 函数中是否使用了 OutputCollector 的实例"输出:output.collect(键，值)这个 - 输出 - 将键值对存储在某处?即使它发送到reducer函数，它们也必须是一个中间文件，对吧?那些文件是什么?它们是否可见并由程序员决定?我们在 main 函数中指定的 OutputKeyClass 和 OutputValueClasses 是这些存储的地方吗?[Text.class 和 IntWritable.class]

I want to know if the OutputCollector's 'instance' output used in the map function: output.collect(key, value) this -output- be storing the key value pairs somewhere? even if it emits to the reducer function, their must be an intermediate file, right? What are those files? Are they visible and decided by the programer? Are the OutputKeyClass, and OutputValueClasses which we specify in the main function these places of storage? [Text.class and IntWritable.class]

我给出了 MapReduce 中 Word Count 示例的标准代码，我们可以在网络的许多地方找到它.

Im giving the standard code for Word Count example in MapReduce, which we can find at many places in the net.

public class WordCount {

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}

public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}

public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);

conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));    
JobClient.runJob(conf);
}
}

推荐答案

Map 函数的输出存储在临时中间文件中.这些文件由 Hadoop 透明处理，因此在正常情况下，程序员无权访问它.如果您对每个映射器内部发生的事情感到好奇，您可以查看相应作业的日志，您可以在其中找到每个映射任务的日志文件.

The output from the Map function is stored in Temporary Intermediate Files. These files are handled transparently by Hadoop, so in a normal scenario, the programmer doesn't have access to that. If you're curious about what's happening inside each mapper, you can review the logs for the respective job where you'll find a log file for each map task.

如果您想控制临时文件的生成位置并访问它们，您必须创建自己的 OutputCollector 类，我不知道这有多容易.

If you want to control where the temporary files are generated, and have access to them, you have to create your own OutputCollector class, and I don't know how easy that is.

如果想看源码，可以使用svn来获取.我认为它可以在这里找到:http://hadoop.apache.org/common/version_control.html.

If you want to have a look at the source code, you can use svn to get it. I think it is available here: http://hadoop.apache.org/common/version_control.html.

这篇关于Hadoop:在 MapReduce 期间 OutputCollector 如何工作?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！