在 oozie 中覆盖 hadoop 的 mapreduce.fileoutputcommitter.marksuccessfuljobs [英] Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie

问题描述

<property>
<name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name>
<value>false</value>
</property>

我想将上述属性覆盖为 true.对于集群上的其余作业，该属性需要为 false，但在我的 oozie 工作流中，我需要在作业完成后使用 hadoop 在输出目录中创建 _SUCCESS 文件.它是工作流中写入输出的配置单元操作.请帮忙.

I want to override the above property to true. The property needs to be false for the rest of the jobs on the cluster, but I need, in my oozie workflow, hadoop to create _SUCCESS file in the output directory after the completion of job. Its a hive action in the workflow which writes output. Please help.

推荐答案

不幸的是，Hive 通过设置它自己的 NullOutputComitter 来覆盖此功能:

Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

见

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

看起来您必须手动创建标志.我们已经为此提交了HIVE-3700.

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.

这篇关于在 oozie 中覆盖 hadoop 的 mapreduce.fileoutputcommitter.marksuccessfuljobs的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！