组合器在哪里组合映射器输出 - 在 Map-reduce 作业的 map 阶段或 reduce 阶段? [英] where does combiners combine mapper outputs - in map phase or reduce phase in a Map-reduce job?

问题描述

我的印象是，combiner 就像是作用于本地 map 任务的 reducer，即聚合单个 Map 任务的结果，以减少输出传输的网络带宽.

I was under the impression that combiners are just like reducers that act on the local map task, That is it aggregates the results of individual Map task in order to reduce the network bandwidth for output transfer.

通过阅读 Hadoop- The权威指南第 3 版，我的理解似乎是正确的.

And from reading Hadoop- The definitive guide 3rd edition, my understanding seems correct.

来自第 2 章(第 34 页)

From chapter 2 (page 34)

组合函数许多 MapReduce 作业受到集群上可用带宽的限制，因此尽量减少在 map 和 reduce 任务之间传输的数据是值得的.Hadoop 允许用户指定在 map 输出上运行的组合器函数——组合器函数的输出形成 reduce 函数的输入.由于组合器功能是一种优化，Hadoop 不保证它会为特定的地图输出记录调用多少次(如果有的话).换句话说，调用组合函数零次、一次或多次应该从减速器产生相同的输出.

所以我在字数问题上尝试了以下方法:

So I tried the following on the wordcount problem:

job.setMapperClass(mapperClass);
job.setCombinerClass(reduceClass);
job.setNumReduceTasks(0);

这里是计数器:

14/07/18 10:40:15 INFO mapred.JobClient: Counters: 10
14/07/18 10:40:15 INFO mapred.JobClient:   File System Counters
14/07/18 10:40:15 INFO mapred.JobClient:     FILE: Number of bytes read=293
14/07/18 10:40:15 INFO mapred.JobClient:     FILE: Number of bytes written=75964
14/07/18 10:40:15 INFO mapred.JobClient:     FILE: Number of read operations=0
14/07/18 10:40:15 INFO mapred.JobClient:     FILE: Number of large read operations=0
14/07/18 10:40:15 INFO mapred.JobClient:     FILE: Number of write operations=0
14/07/18 10:40:15 INFO mapred.JobClient:   Map-Reduce Framework
14/07/18 10:40:15 INFO mapred.JobClient:     Map input records=7
14/07/18 10:40:15 INFO mapred.JobClient:     Map output records=16
14/07/18 10:40:15 INFO mapred.JobClient:     Input split bytes=125
14/07/18 10:40:15 INFO mapred.JobClient:     Spilled Records=0
14/07/18 10:40:15 INFO mapred.JobClient:     Total committed heap usage (bytes)=85000192

这里是part-m-00000:

hello   1
world   1
Hadoop  1
programming 1
mapreduce   1
wordcount   1
lets    1
see 1
if  1
this    1
works   1
12345678    1
hello   1
world   1
mapreduce   1
wordcount   1

显然没有应用组合器.我知道 Hadoop 不保证是否会调用组合器.但是当我打开 reduce 阶段时，组合器会被调用.

so clearly no combiner is applied. I understand that Hadoop does not guarantee if a combiner will be called at all. But when I turn on the reduce phase, the combiner gets called.

为什么会这样?

现在，当我阅读第 6 章(第 208 页)关于MapReduce 的工作原理 时.我在Reduce side中看到了这段描述.

Now when I read chapter 6 (page 208) on how MapReduce works. I see this paragraph described in the Reduce side.

如果映射输出足够小，则将其复制到 reduce 任务 JVM 的内存(缓冲区的大小由 mapred.job.shuffle.input.buffer.percent 控制，它指定要使用的堆的比例以此目的);否则，它们将被复制到磁盘.当内存缓冲区达到阈值大小(由 mapred.job.shuffle.merge.percent 控制)或达到映射输出的阈值数量(mapred.inmem.merge.threshold)时，它被合并并溢出到磁盘.如果指定了组合器，它将在合并期间运行以减少写入磁盘的数据量.

我从这一段的推论是:1) Combiner ALSO 在 reduce 阶段运行.

My inferences from this paragraph are : 1) Combiner is ALSO run during the reduce phase.

推荐答案

combiner的主要功能是优化.在大多数情况下，它就像一个小型减速器.来自同一本书的第 206 页，章节 - mapreduce 的工作原理(地图方面):

The main function of a combiner is optimization. It acts like a mini-reducer for most cases. From page 206 of the same book, chapter - How mapreduce works(The map side):

运行 combiner 函数可以得到更紧凑的 map 输出，因此写入本地磁盘和传输到 reducer 的数据更少.

Running the combiner function makes for a more compact map output, so there is less data to write to local disk and to transfer to the reducer.

引用你的问题，

如果指定了组合器，它将在合并期间运行以减少写入磁盘的数据量.

If a combiner is specified it will be run during the merge to reduce the amount of data written to disk.

两个引号都表明 combiner 主要是为了紧凑而运行.减少输出传输的网络带宽是此优化的一个优势.

Both the quotes indicate that a combiner is run primarily for compactness. Reducing the network bandwidth for output transfer is an advantage of this optimization.

另外，来自同一本书，

回想一下组合器可以在输入上重复运行而不影响最终结果.如果只有一两次溢出，那么地图输出大小的潜在减少是不值得的调用组合器的开销，因此不会针对此地图输出再次运行.

Recall that combiners may be run repeatedly over the input without affecting the final result. If there are only one or two spills, then the potential reduction in map output size is not worth the overhead in invoking the combiner, so it is not run again for this map output.

意味着 hadoop 不保证组合器运行多少次(也可以为零)

Meaning that hadoop doesn't guarentee how many times a combiner is run(could be zero also)

永远不会为仅地图作业运行组合器.这是有道理的，因为组合器会更改地图输出.此外，由于它不保证调用的次数，因此也不保证映射输出相同.

A combiner is never run for map-only jobs. It makes sense because, a combiner changes the map output. Also, since it doesn't guarantee the number of times it is called, the map output is not guaranteed to be the same either.

这篇关于组合器在哪里组合映射器输出 - 在 Map-reduce 作业的 map 阶段或 reduce 阶段?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！