在oozie中覆盖hadoop的mapreduce.fileoutputcommitter.marksuccessfuljobs [英] Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie

问题描述

<property>
<name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name>
<value>false</value>
</property>

我想覆盖上述属性为true。该属性对于群集中的其余作业需要为false，但在我的oozie工作流程中，需要hadoop在作业完成后在输出目录中创建_SUCCESS文件。它是写入输出的工作流程中的配置单元操作。

I want to override the above property to true. The property needs to be false for the rest of the jobs on the cluster, but I need, in my oozie workflow, hadoop to create _SUCCESS file in the output directory after the completion of job. Its a hive action in the workflow which writes output. Please help.

推荐答案

Hive不幸地通过设置它自己的NullOutputComter来覆盖此功能：

Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

见

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

看起来您必须手动创建标志。我们已经为此提交了 HIVE-3700 。

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.

这篇关于在oozie中覆盖hadoop的mapreduce.fileoutputcommitter.marksuccessfuljobs的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！