如何分辨MapReduce使用多少个映射器？ [英] How to tell MapReduce how many mappers to use?

问题描述

有没有什么办法可以告诉hadoop使用特定数量的mapper / reducer进程？或者，至少是最少的mapper进程？

在文档中指出，您可以使用方法来做到这一点

  public void setNumMapTasks（int n）
  

< JobConf类的一部分。

这种方式并没有过时，所以我开始使用Job类的Job。这样做的正确方法是什么？

解决方案

地图任务的数量由输入中的块数决定。如果输入文件为100MB，并且HDFS块大小为64MB，则输入文件将占用2个块。因此，将会产生2个地图任务。 JobConf.setNumMapTasks（）（1）提示框架。

减法器的数量由JboConf.setNumReduceTasks（）函数设置。这确定了作业的减少任务的总数。另外，mapred.tasktracker.tasks.maximum参数确定可以在单个作业跟踪器节点上并行运行的reduce任务的数量。

你可以在这里找到更多关于地图数量和减少工作的信息（2）

（1 ） - http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapred/JobConf.html#setNumMapTasks%28int%29

（ 2） - http://wiki.apache.org/hadoop/HowManyMapsAndReduces

I am trying to speed optimize MapReduce job.

Is there any way I can tell hadoop to use a particular number of mapper/reducer processes? Or, at least, minimal number of mapper processes?

In the documentation, it is specified, that you can do that with the method

public void setNumMapTasks(int n)

of the JobConf class.

That way is not obsolete, so I am starting the Job with Job class. What is the right way of doing this?

解决方案

The number of map tasks is determined by the number of blocks in the input. If the input file is 100MB and the HDFS block size is 64MB then the input file will take 2 blocks. So, 2 map tasks will be spawned. JobConf.setNumMapTasks() (1) a hint to the framework.

The number of reducers is set by the JboConf.setNumReduceTasks() function. This determines the total number of reduce tasks for the job. Also, the mapred.tasktracker.tasks.maximum parameter determines the number of reduce tasks which can run parallely on a single job tracker node.

You can find more information here on the number of map and reduce jobs at (2)

(1) - http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapred/JobConf.html#setNumMapTasks%28int%29
(2) - http://wiki.apache.org/hadoop/HowManyMapsAndReduces

这篇关于如何分辨MapReduce使用多少个映射器？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！