如何在Hadoop 3.0中做CopyMerge? [英] How to do CopyMerge in Hadoop 3.0?
问题描述
我知道 hadoop 版本
2.7 的
FileUtil
有 copyMerge
函数可将多个文件合并为一个新文件。
但 copyMerge在
函数。 3.0
版本中,每个API不再支持
有关如何将目录中的所有文件合并到 3.0
版本的hadoop中的新单个文件的任何想法?
FileUtil#copyMerge方法已被删除。查看主要更改的详细信息:
https://issues.apache.org/jira/browse/HADOOP-12967
https://issues.apache.org/jira/browse/HADOOP-11392
您可以使用getmerge
用法:hadoop fs -getmerge [-nl]
源目录和目标文件作为输入,并将src中的文件连接到目标本地文件。可选地,可以将-nl设置为在每个文件的末尾添加换行符(LF)。 -skip-empty-file可以用来避免空文件中不需要的换行符。
示例:
hadoop fs -getmerge -nl / src /opt/output.txt
hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt / output。 txt
退出代码:
成功时返回0,错误时返回非零。 p>
I know hadoop
version 2.7
's FileUtil
has the copyMerge
function that merges multiple files into a new one.
But the copyMerge
function is no longer supported per the API in the 3.0
version.
Any ideas on how to merge all files within a directory into a new single file in the 3.0
version of hadoop?
FileUtil#copyMerge method has been removed. See details for the major change:
https://issues.apache.org/jira/browse/HADOOP-12967
https://issues.apache.org/jira/browse/HADOOP-11392
You can use getmerge
Usage: hadoop fs -getmerge [-nl]
Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally -nl can be set to enable adding a newline character (LF) at the end of each file. -skip-empty-file can be used to avoid unwanted newline characters in case of empty files.
Examples:
hadoop fs -getmerge -nl /src /opt/output.txt
hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt /output.txt
Exit Code: Returns 0 on success and non-zero on error.
这篇关于如何在Hadoop 3.0中做CopyMerge?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!