如何将数据附加到现有的镶木地板文件 [英] How to append data to an existing parquet file

问题描述

我正在使用以下代码创建 ParquetWriter 并向其写入记录.

I'm using the following code to create ParquetWriter and to write records to it.

ParquetWriter<GenericRecord> parquetWriter = new ParquetWriter(path, writeSupport, CompressionCodecName.SNAPPY, BLOCK_SIZE, PAGE_SIZE);

final GenericRecord record = new GenericData.Record(avroSchema);

<小时>

parquetWriter.write(record);

但它只允许创建新文件(在指定的路径).有没有办法将数据附加到现有的镶木地板文件(在路径)?在我的情况下，缓存 parquetWriter 是不可行的.

But it only allows to create new files(at the specfied path). Is there a way to append data to an existing parquet file (at path)? Caching parquetWriter is not feasible in my case.

推荐答案

有一个 Spark API SaveMode 叫做 append:https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/SaveMode.html 我相信这可以解决您的问题.

There is a Spark API SaveMode called append: https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/SaveMode.html which I believe solves your problem.

使用示例:

df.write.mode('append').parquet('parquet_data_file')

这篇关于如何将数据附加到现有的镶木地板文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！