Hadoop MapReduce 中 Mapper/Reducer 的设置和清理方法 [英] setup and cleanup methods of Mapper/Reducer in Hadoop MapReduce

问题描述

是否分别在每个 mapper 和 reducer 任务中调用 setup 和 cleanup 方法?还是在整个映射器和减速器作业开始时只调用一次?

Are setup and cleanup methods called in each mapper and reducer tasks respectively? Or are they called only once at the start of overall mapper and reducer jobs?

推荐答案

每个任务都会调用它们，所以如果你有 20 个正在运行的映射器，则会为每个任务调用 setup/cleanup.

They are called for each task, so if you have 20 mappers running, the setup / cleanup will be called for each one.

一个问题是 Mapper 和 Reducer 的标准 run 方法不会捕获 map/reduce 方法周围的异常 - 因此，如果在这些方法中抛出异常，则不会调用 clean up 方法.

2020 年编辑:如评论中所述，2012 年 (Hadoop 0.20) 的这一说法不再正确，清理被称为 finally 块的一部分.

2020 Edit: As noted in the comments, this statement from 2012 (Hadoop 0.20) is no longer true, the cleanup is called as part of a finally block.

这篇关于Hadoop MapReduce 中 Mapper/Reducer 的设置和清理方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！