如何限制映射器的数量 [英] how to limit the number of mappers

问题描述

我在我的 java 程序中使用 conf.setNumMapTasks() 明确指定了映射器的数量，但是当作业结束时，计数器显示启动的映射任务的数量超过了指定的值.如何将mapper的数量限制为指定值?

I explicitly specify the number of mappers within my java program using conf.setNumMapTasks(), but when the job ends, the counter shows that the number of launched map tasks were more than the specified value. How to limit the number of mapper to the specified value?

推荐答案

根据Hadoop API Jonf.setNumMapTasks 只是对 Hadoop 运行时的提示.map任务的总数等于输入数据中要处理的块数.

According to the Hadoop API Jonf.setNumMapTasks is just a hint to the Hadoop runtime. The total number of map tasks equals to the number of blocks in the input data to be processed.

虽然，应该可以使用 mapred.tasktracker.map.tasks.maximum 和 mapred.tasktracker.reduce 配置每个节点的 map/reduce 槽数.mapred-site.xml 中的 tasks.maximum.这样就可以配置在整个集群中并行执行的映射器/减速器的总数.

Although, it should be possible to configure the number of map/reduce slots per node by using the mapred.tasktracker.map.tasks.maximum and the mapred.tasktracker.reduce.tasks.maximum in mapred-site.xml. This way it's possible to configure the total number of mappers/reducers executing in parallel across the entire cluster.

这篇关于如何限制映射器的数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！