Hadoop中没有Reducer的组合器 [英] Combiner without Reducer in Hadoop

问题描述

我可以写一个只有Mappers和Combiners（即没有Reducer的mini-redurs）的Hadoop代码吗？

job.setMapperClass （WordCountMapper.class）;
job.setCombinerClass（WordCountReducer.class）;

conf.setInt（mapred.reduce.tasks，0 ）;

我试图这样做，但我总是看到我在作业跟踪链接上有一个减少任务

启动reduce任务= 1

如何删除还原剂，同时保持合成器？这是可能的吗？

解决方案

在你描述的情况下你应该使用Reducers。用作键：Context.getInputSplit（）。getPath（）+ Context.getInputSplit（）。getStart（） - 这个组合对于每个Mapper都是唯一的。

Can I write a Hadoop code that has only Mappers and Combiners (i.e. mini-reducers with no reducer)?

job.setMapperClass(WordCountMapper.class);
job.setCombinerClass(WordCountReducer.class);

conf.setInt("mapred.reduce.tasks", 0);

I was trying to do so but I always see that I have one reduce task on the job tracker link

Launched reduce tasks = 1

How can I delete reducers while keeping combiners? is that possible?

解决方案

In the case you describe you should use Reducers. Use as key: Context.getInputSplit().getPath() + Context.getInputSplit().getStart() - this combination is unique for each Mapper.

这篇关于Hadoop中没有Reducer的组合器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！