Hadoop MapReduce中Mapper / Reducer的设置和清理方法 [英] setup and cleanup methods of Mapper/Reducer in Hadoop MapReduce

问题描述

是否在每个映射器和减速器任务中分别调用设置和清理方法？还是他们在整个映射器和减速器工作开始时只召唤过一次？

解决方案

每个任务都会调用它们，所以如果您有20个映射器在运行，则会为每个映射调用setup / cleanup 。

一个问题是Mapper和Reducer的标准运行方法不能捕获map / reduce方法的异常 - 所以如果在这些方法中引发异常，清理方法不会被调用。

Are setup and cleanup methods called in each mapper and reducer tasks respectively? Or are they called only once at the start of overall mapper and reducer jobs?

解决方案

They are called for each task, so if you have 20 mappers running, the setup / cleanup will be called for each one.

One gotcha is the standard run method for both Mapper and Reducer does not catch exceptions around the map / reduce methods - so if an exception is thrown in these methods, the clean up method will not be called.

这篇关于Hadoop MapReduce中Mapper / Reducer的设置和清理方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！