在减少阶段后合并输出文件 [英] merge output files after reduce phase

问题描述

在mapreduce 中，每个reduce 任务将其输出写入一个名为part-r-nnnnn 的文件，其中nnnnn 是与reduce 任务关联的分区ID.map/reduce 是否合并这些文件?如果是，如何?

In mapreduce each reduce task write its output to a file named part-r-nnnnn where nnnnn is a partition ID associated with the reduce task. Does map/reduce merge these files? If yes, how?

推荐答案

您可以通过调用以下方式委托reduce输出文件的整个合并，而不是自己进行文件合并:

Instead of doing the file merging on your own, you can delegate the entire merging of the reduce output files by calling:

hadoop fs -getmerge /output/dir/on/hdfs/ /desired/local/output/file.txt

注意这会在本地合并 HDFS 文件.运行前确保你有足够的磁盘空间

Note This combines the HDFS files locally. Make sure you have enough disk space before running

这篇关于在减少阶段后合并输出文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！