hadoop：还原剂的数量保持不变4 [英] hadoop: number of reducers remains a constant 4

问题描述

我使用 mapred.reduce.tasks = 100 来运行hadoop作业（只是试验）。产生的地图数量为537，因为这取决于输入分割。问题是减速器并行运行的次数不会超过4.即使在映射完成后100％。有没有一种方法可以增加减速器的运行数量，因为CPU使用率是次优的，Reduce速度非常慢。

我还设置了 mapred.tasktracker.reduce.tasks.maximum = 100 。但这似乎并不影响并行运行的还原器的数量。

解决方案

事实证明，所有这一切都是需要的在更改mapred-site.xml之后重新启动mapred和dfs守护进程。 mapred.tasktracker.reduce.tasks.maximum 确实是增加Reduce容量的正确参数。

无法理解为什么每次提交作业时，hadoop都选择不重新加载 mapred-site 。

I'm running a hadoop job with mapred.reduce.tasks = 100 (just experimenting). The number of maps spawned are 537 as that depends on the input splits. Problem is the number of reducers "Running" in parallel won't go beyond 4. Even after the maps are 100% complete. Is there a way to increase the number of reducers running as the CPU usage is sub optimal and the Reduce is very slow.

I have also set mapred.tasktracker.reduce.tasks.maximum = 100. But this doesn't seem to affect the numbers of reducers running in parallel.

解决方案

It turns out all that was required was a restart of the mapred and dfs daemons after you change the mapred-site.xml. mapred.tasktracker.reduce.tasks.maximum is indeed the right parameter to be set to increase the Reduce capacity.

Can't understand why hadoop chose not to reload the mapred-site every time when a job is submitted.

这篇关于hadoop：还原剂的数量保持不变4的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！