组合器实施和内部工作 [英] Combiner Implementation and internal working

问题描述

我想在我的 MR 代码中使用组合器，比如 WordCount.

I want to use a combiner in my MR code say WordCount.

我应该如何实现它?

什么样的数据从组合器传递给reducer?

What sort of data is being passed to the reducer from the combiner?

如果你们中的任何人都可以提供Combiner 和Reducer 的代码，那就太好了.

It will be great if anyone of you can provide me codes of both Combiner as well as the Reducer.

如果你能解释一下组合器的工作原理会更好

我是 mapreduce 的新手，我正处于学习阶段.

I am new to mapreduce and I am at a learning stage.

提前致谢:)

推荐答案

一个Combiner，也称为半减速器.

A Combiner, also known as a semi-reducer.

Combiner的主要作用是对同一个key的map输出记录进行汇总.

The main function of a Combiner is to summarize the map output records with the same key.

Combiner 类用于 Map 类和 Reduce 类之间，以减少 Map 和 Reduce 之间的数据传输量

The Combiner class is used in between the Map class and the Reduce class to reduce the volume of data transfer between Map and Reduce

示例代码说明.

地图输入:

What do you mean by Object
What do you know about Java
What is Java Virtual Machine
How Java enabled High Performance

地图输出

<What,1> <do,1> <you,1> <mean,1> <by,1> <Object,1>
<What,1> <do,1> <you,1> <know,1> <about,1> <Java,1>
<What,1> <is,1> <Java,1> <Virtual,1> <Machine,1>
<How,1> <Java,1> <enabled,1> <High,1> <Performance,1>

此 MAP 输出将作为输入传递给组合器.

This MAP output will be passed as input to Combiner.

组合器输出

   <What,1,1,1> <do,1,1> <you,1,1> <mean,1> <by,1> <Object,1>
   <know,1> <about,1> <Java,1,1,1>
   <is,1> <Virtual,1> <Machine,1>
   <How,1> <enabled,1> <High,1> <Performance,1>

此组合器输出作为输入传递给 Reducer.

This combiner output is passed as input to Reducer.

减速器输出

   <What,3> <do,2> <you,2> <mean,1> <by,1> <Object,1>
   <know,1> <about,1> <Java,3>
   <is,1> <Virtual,1> <Machine,1>
   How,1> <enabled,1> <High,1> <Performance,1> 

如果你使用java，下面的代码将设置Combiner &减速器到同一类，这是理想的.

If you are using java, below code will set Combiner & Reducer to same class, which is ideal.

  job.setJarByClass(WordCount.class);
  job.setMapperClass(TokenizerMapper.class);
  job.setCombinerClass(IntSumReducer.class);
  job.setReducerClass(IntSumReducer.class);

看看在 java @tutorialspoint 中工作的 example

Have a look at working example in java @tutorialspoint

这篇关于组合器实施和内部工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！