是否有可能有一个map-reduce的多个输出文件？ [英] Is it possible to have multiple output files for a map-reduce?

问题描述

在我的输入文件中，我有一个列作为国家。现在，我的任务是将某个国家的记录放入与该国家名称相同的单独文件中。这可能在Map-reduce中做到。
请分享您的想法。

解决方案

如果您使用新的API，您应该查看< a href =http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html =nofollow> MultipleOutputs class 。

提交作业的使用模式：

 
 
 
 
工作职位=新职位（）; 
 
 FileInputFormat.setInputPath（job，inDir）; 
 FileOutputFormat.setOutputPath（job，outDir）; 
 
 job.setMapperClass（MOMap.class）; 
 job.setReducerClass（MOReduce.class）; 
 ... 
 
 //为作业定义额外的单个文本输出'text'
 MultipleOutputs.addNamedOutput（job，text，TextOutputFormat.class，
 LongWritable.class，Text.class）; 
 
 //为作业
定义附加的基于序列文件的输出'序列'MultipleOutputs.addNamedOutput（job，seq，
 SequenceFileOutputFormat.class，
 LongWritable .class，Text.class）; 
 ... 
 
 job.waitForCompletion（true）; 
 ... 
 
 

Reducer的用法：

< pre

$ b String generateFileName（K k，V v）{
return k.toString（）+_+ v.toString（）;
}

public class MOReduce扩展
Reducer {
private MultipleOutputs mos;
public void setup（Context context）{
...
mos = new MultipleOutputs（context）;

$ b $ public void reduce（WritableComparable key，Iterator values，
Context context）
抛出IOException {
...
mos。写（文本，，键，新的文本（你好））;
mos.write（seq，LongWritable（1），new Text（Bye），seq_a）;
mos.write（seq，LongWritable（2），key，new Text（Chau），seq_b）;
mos.write（key，new Text（value），generateFileName（key，new Text（value）））;
...
}

public void cleanup（Context）throws IOException {
mos.close（）;
...
}
}

In my input file i have a column as country. Now, my task is to place records of a particular country into a separate file naming with that country. Is this possible to do in Map-reduce.! Please share your ideas regarding this.

解决方案

If you are using the new API , you should look at the MultipleOutputs class. There is an example inside this class.

Usage pattern for job submission:

    Job job = new Job();

    FileInputFormat.setInputPath(job, inDir);
    FileOutputFormat.setOutputPath(job, outDir);

    job.setMapperClass(MOMap.class);
    job.setReducerClass(MOReduce.class);
    ...

    // Defines additional single text based output 'text' for the job
    MultipleOutputs.addNamedOutput(job, "text", TextOutputFormat.class,
    LongWritable.class, Text.class);

    // Defines additional sequence-file based output 'sequence' for the job
    MultipleOutputs.addNamedOutput(job, "seq",
      SequenceFileOutputFormat.class,
      LongWritable.class, Text.class);
    ...

    job.waitForCompletion(true);
    ...

Usage in Reducer:

    String generateFileName(K k, V v) {
       return k.toString() + "_" + v.toString();
    }

    public class MOReduce extends
       Reducer {
         private MultipleOutputs mos;
         public void setup(Context context) {
          ...
              mos = new MultipleOutputs(context);
          }

         public void reduce(WritableComparable key, Iterator values,
                Context context)
                throws IOException {
          ...
     mos.write("text", , key, new Text("Hello"));
     mos.write("seq", LongWritable(1), new Text("Bye"), "seq_a");
     mos.write("seq", LongWritable(2), key, new Text("Chau"), "seq_b");
     mos.write(key, new Text("value"), generateFileName(key, new Text("value")));
     ...
      }

    public void cleanup(Context) throws IOException {
         mos.close();
     ...
    }
   }

这篇关于是否有可能有一个map-reduce的多个输出文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！