如何在 Hadoop 1.0.4 中链接映射器/减速器? [英] How to chain mapper/reducer in Hadoop 1.0.4?

问题描述

我使用的是 Hadoop 1.0.4 的新"API(org.apache.hadoop.mapreduce 包中的类).当我想链接 mapper/reducer 时，我发现 ChainMapper、ChainReducer 是为旧"API(包 org.apache.hadoop.mapred 中的类)编写的.我该怎么办?

I was using "new" API of Hadoop 1.0.4 (classes in package org.apache.hadoop.mapreduce). When I wanted to chain mapper/reducer, I found out that ChainMapper, ChainReducer are written for the "old" API (classes in package org.apache.hadoop.mapred). What should I do?

推荐答案

我也在寻找同样的问题.我确实得到了答案，尽管为时已晚，但我认为分享这可能会对某人有所帮助.

I was also searching for the same. I did get the answer and even though its late I thought sharing this may help someone.

从 Hadoop 2.0 开始，您可以在 org.apache.hadoop.mapreduce.lib.chain

From Hadoop 2.0 onwards you can find ChainMapper and ChainReducer in the package org.apache.hadoop.mapreduce.lib.chain

...
Job job = new Job(conf, "MyJob");

Configuration map1Conf = new Configuration(false); 
... ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf);

Configuration map2Conf = new Configuration(false); 
... ChainMapper.addMapper(job, BMap.class, Text.class, Text.class, LongWritable.class, Text.class, false, map2Conf);

Configuration map3Conf = new Configuration(false); 
... ChainReducer.setReducer(job, CReducer.class, Text.class, Text.class, LongWritable.class, Text.class, false, map3Conf);
...

job.waitForComplettion(true);
... 

这篇关于如何在 Hadoop 1.0.4 中链接映射器/减速器?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！