是否可以使用Snappy压缩将Avro文件加载到BigQuery中？ [英] Is it possible to load Avro files with Snappy compression into BigQuery?

问题描述

我知道BigQuery支持Avro文件上传，并且我成功地将Avro文件加载到BigQuery中。

使用以下命令，

  java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json> Output.snappy.avro 
  

我用Snappy压缩生成了Avro文件并尝试加载到BigQuery中，但加载作业失败，出现以下错误，

 错误：
 file-00000000：Apache Avro库无法解析文件文件-00000000。 （错误代码：无效）
  

是否可以使用Snappy压缩将Avro文件加载到BigQuery中？

解决方案

BigQuery仅支持Avro数据块压缩的DEFLATE和Snappy算法，来自文档（ https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro ）

压缩的Avro文件不受支持，但是压缩的数据块
是。 BigQuery支持DEFLATE和Snappy编解码器。

I know that BigQuery supports Avro file upload and I'm successful in loading Avro file into BigQuery.

Using below command,

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro

I have generated an Avro file with Snappy compression and trying to load into BigQuery but Load job fails with below errors,

Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)

Is it possible to load Avro files with Snappy compression into BigQuery?

解决方案

BigQuery only supports DEFLATE and Snappy algorithm for Avro data blocks compression, from the docs (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro)

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.

这篇关于是否可以使用Snappy压缩将Avro文件加载到BigQuery中？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！