Hadoop MapReduce中间输出 [英] Hadoop MapReduce intermediate output

问题描述

有没有办法输出以记录MapReduce作业的中间（Map Phase）输出而不编辑应用程序？ （该应用程序不是我的，但集群是，我可以设置Hadoop集群，因为我想要）

一个href =http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms =noreferrer> keep.task.files.pattern 参数可以用来保存中间文件。一旦作业完成，中间文件必须手动清理。因为这是一个map / reduce任务属性，所以必须在配置文件中设置它，并再次打包jar文件。

Is there a way to output to log the intermediate (Map Phase) output of a MapReduce Job without editing the Application? (The application is not mine, but the cluster is, and I can setup the Hadoop Cluster as I want to)

解决方案

keep.task.files.pattern parameter can be used to keep the intermediate files. The intermediate files have to be manually cleaned up once the Job has been completed. Since, this is a map/reduce task property, it has to be set in the configuration file and the jar file packaged again.

这篇关于Hadoop MapReduce中间输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！