在Hadoop / PIG中压缩/解压gzip数据是否透明？ [英] Is compression/decompression of gzip data transparent in Hadoop/PIG?

问题描述

我在某处读到Hadoop内置了对压缩和解压缩的支持，但我想这是关于mapper输出（通过设置一些属性）？

我想知道是否有任何特定的PIG加载/存储功能可用于读取压缩数据或以压缩格式输出数据？

解决方案

PigStorage通过检查文件名来处理压缩输入：

• *。bz2 / * .bz - org.apache.pig.bzip2r.Bzip2TextInputFormat


• else使用 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat
  - 这扩展了 oahmapreduce.TextinputFormat 它可以处理.gz和zippy文件，如果你已经安装了编解码器
  
  
  

  输出通过一些属性处理：

/ p>




• output.compression.enabled - true / false $ b $ （ org.apache.hadoop.io.compress.GzipCodec / code> for gzip）






如果您对此感兴趣，可以通过PigStorage.java进行挖掘对你的兴趣


I read somewhere that Hadoop has a built-in support for compression and decompression but I guess it is about mapper output (by setting some properties)?

I wonder if there is any particular PIG load/store functions I can use for reading compressed data or outputting data as compressed?

解决方案

The PigStorage handles compressed input by examining the file names:

• *.bz2 / *.bz - org.apache.pig.bzip2r.Bzip2TextInputFormat
• Everything else uses org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat -- This extends o.a.h.mapreduce.TextinputFormat which can handle .gz and zippy files if you have the codecs installed

Output is handled via some properties:

• output.compression.enabled - true / false
• output.compression.codec - the class name of the codec to use (org.apache.hadoop.io.compress.GzipCodec for gzip)

If you're feeling up to it, digging through the PigStorage.java may be of interest to you

这篇关于在Hadoop / PIG中压缩/解压gzip数据是否透明？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！