Hadoop：如何防止失败的任务使整个工作失败？ [英] Hadoop: How can I prevent failed tasks from making the whole job fail?

问题描述

我正在运行一个hadoop工作，比如说1000个任务。我需要这份工作来尝试运行每项任务，但许多任务不会完成，而是会抛出异常。我无法改变这种行为，但我仍然需要从没有失败的任务中获得的数据。

我怎样才能确保Hadoop完成所有1000个任务遇到大量失败的任务？

解决方案

在您的情况下，您可以设置允许失败的任务的最大百分比而不会触发工作失败。使用

  mapred.max.map.failures.percent 
 mapred来独立控制Map任务和减少任务。 max.reduce.failures.percent 
  

属性。所以如果你想要70％的任务结果，即使30％失败，你也可以用上面的属性来完成。

I'm running a hadoop job with, say, 1000 tasks. I need the job to attempt to run every task but many of the tasks will not complete and will instead throw an exception. I cannot change this behavior, but I still need the data obtained from the tasks that did not fail.

How can I make sure Hadoop goes through with all the 1000 tasks despite encountering a large number of failed tasks?

解决方案

In your case, you could set the maximum percentage of tasks that are allowed to fail without triggering job failure. Map tasks and reduce tasks are controlled independently, using the

mapred.max.map.failures.percent 
mapred.max.reduce.failures.percent 

properties. So if you want 70% of tasks result even if 30% fails you could do so with above properties.

这篇关于Hadoop：如何防止失败的任务使整个工作失败？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！