如何在映射器（或缩减器）内放弃MR作业 [英] How to abort a MR job inside a mapper (or a reducer)

问题描述

我试图在映射方法中抛出IOExceptions，但MR作业没有停止。抛出大量IOException后，该作业将停止。
是否有任何方法通过抛出一个异常或一些简单的调用来停止整个工作？
Thanks。

I tried to throw IOExceptions in the map method but the MR job is not stopped. The job will be stopped after tons of IOException thrown. Is there any way to stop the whole job by just throwing an exception or some simple calls? Thanks.

推荐答案

这不是Hadoop的理想用例，也不是一个好习惯，但您可以在你的代码中杀死你的工作。
因此，无论何时您达到您希望停止工作的条件时，请记录需求并杀死您的工作。

This isn't an ideal use-case for Hadoop and not a good practice but you can kill your job right from inside your code. So whenever you reach the condition where in you want your job to stop, log the needful and kill you job.

这可以通过RunningJob.killjob ）使用旧的 mapred API或使用 Job.killJob（）。您应该在 configure（）中为jobID获取对 RunningJob 或 Job code>或 setup（）。然后在需要时调用kill作业，新API的伪代码如下所示：



This can be done RunningJob.killjob() using the old mapred API or by using Job.killJob(). You should get reference to RunningJob or Job objects for the jobID in configure() or setup() respectively. And then call the kill job when you need, a pseudo code for new API would look as follows:

Class Map extends mapper<K1,V1,K2,V2>{
Job myJob;
@Override
setup(){
// Get the JObID
// Get the Job object
}

map(){
...
if(condition-to-stop){
myJob.killJob();
...
}
}
}

这篇关于如何在映射器（或缩减器）内放弃MR作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！