如何解压缩hadoop减少输出文件以快速结束？ [英] How to decompress the hadoop reduce output file end with snappy?

问题描述

我们的hadoop集群使用snappy作为默认编解码器。 Hadoop作业减少输出文件名称就像 part-r-00000.snappy 。 JSnappy无法解压文件bcz JSnappy要求文件以SNZ开头。减少输出文件以某种方式从一些字节0开始。

如何解压文件？

使用Hadoop fs -text来读取该文件并将其传送到txt文件。
ex：

hadoop fs -text part-r-00001.snappy> /tmp/mydatafile.txt

Our hadoop cluster using snappy as default codec. Hadoop job reduce output file name is like part-r-00000.snappy. JSnappy fails to decompress the file bcz JSnappy requires the file start with SNZ. The reduce output file start with some bytes 0 somehow.

How could I decompress the file?

解决方案

Use "Hadoop fs -text" to read this file and pipe it to txt file. ex:

hadoop fs -text part-r-00001.snappy > /tmp/mydatafile.txt

这篇关于如何解压缩hadoop减少输出文件以快速结束？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！