如何在映射器(或减速器)中中止 MR 作业 [英] How to abort a MR job inside a mapper (or a reducer)

问题描述

我尝试在 map 方法中抛出 IOExceptions，但 MR 作业没有停止.在抛出大量 IOException 后，该作业将停止.有没有办法通过抛出异常或一些简单的调用来停止整个工作?谢谢.

I tried to throw IOExceptions in the map method but the MR job is not stopped. The job will be stopped after tons of IOException thrown. Is there any way to stop the whole job by just throwing an exception or some simple calls? Thanks.

推荐答案

这不是 Hadoop 的理想用例，也不是一个好的实践，但您可以直接从代码内部杀死您的工作.因此，当您达到希望停止工作的条件时，记录必要的并终止您的工作.

This isn't an ideal use-case for Hadoop and not a good practice but you can kill your job right from inside your code. So whenever you reach the condition where in you want your job to stop, log the needful and kill you job.

这可以使用旧的 mapred API 或使用 Job.killJob() 来实现 RunningJob.killjob().您应该分别在 configure() 或 setup() 中获得对 jobID 的 RunningJob 或 Job 对象的引用.然后在需要时调用 kill 作业，新 API 的伪代码如下所示:

This can be done RunningJob.killjob() using the old mapred API or by using Job.killJob(). You should get reference to RunningJob or Job objects for the jobID in configure() or setup() respectively. And then call the kill job when you need, a pseudo code for new API would look as follows:

Class Map extends mapper<K1,V1,K2,V2>{
Job myJob;
@Override
setup(){
// Get the JObID
// Get the Job object
}

map(){
...
if(condition-to-stop){
myJob.killJob();
...
}
}
}

这篇关于如何在映射器(或减速器)中中止 MR 作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！