Hadoop MapReduce 中间输出 [英] Hadoop MapReduce intermediate output

问题描述

有没有办法在不编辑应用程序的情况下输出以记录 MapReduce 作业的中间(映射阶段)输出?(应用程序不是我的，但集群是，我可以根据需要设置 Hadoop 集群)

Is there a way to output to log the intermediate (Map Phase) output of a MapReduce Job without editing the Application? (The application is not mine, but the cluster is, and I can setup the Hadoop Cluster as I want to)

推荐答案

keep.task.files.pattern 参数可用于保留中间文件.作业完成后，必须手动清理中间文件.由于这是一个map/reduce任务属性，所以必须在配置文件中设置并重新打包jar文件.

keep.task.files.pattern parameter can be used to keep the intermediate files. The intermediate files have to be manually cleaned up once the Job has been completed. Since, this is a map/reduce task property, it has to be set in the configuration file and the jar file packaged again.

这篇关于Hadoop MapReduce 中间输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！