gzip 数据的压缩/解压缩在 Hadoop/PIG 中是否透明? [英] Is compression/decompression of gzip data transparent in Hadoop/PIG?

问题描述

我在某处读到 Hadoop 内置了对压缩和解压缩的支持，但我想这是关于映射器输出(通过设置一些属性)?

I read somewhere that Hadoop has a built-in support for compression and decompression but I guess it is about mapper output (by setting some properties)?

我想知道是否有任何特定的 PIG 加载/存储函数可以用于读取压缩数据或输出压缩数据?

I wonder if there is any particular PIG load/store functions I can use for reading compressed data or outputting data as compressed?

推荐答案

PigStorage 通过检查文件名来处理压缩输入:

The PigStorage handles compressed input by examining the file names:

• *.bz2/*.bz - org.apache.pig.bzip2r.Bzip2TextInputFormat
• 其他一切都使用 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat-- 这扩展了 o.a.h.mapreduce.TextinputFormat，如果您安装了编解码器，它可以处理 .gz 和 zippy 文件

• *.bz2 / *.bz - org.apache.pig.bzip2r.Bzip2TextInputFormat
• Everything else uses org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat -- This extends o.a.h.mapreduce.TextinputFormat which can handle .gz and zippy files if you have the codecs installed

输出是通过一些属性处理的:

Output is handled via some properties:

• output.compression.enabled - 真/假
• output.compression.codec - 要使用的编解码器的类名(org.apache.hadoop.io.compress.GzipCodec for gzip)

• output.compression.enabled - true / false
• output.compression.codec - the class name of the codec to use (org.apache.hadoop.io.compress.GzipCodec for gzip)

如果您愿意，深入研究 PigStorage.java 可能对您感兴趣

If you're feeling up to it, digging through the PigStorage.java may be of interest to you

这篇关于gzip 数据的压缩/解压缩在 Hadoop/PIG 中是否透明?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！