pandas df.to_parquet 写入多个较小的文件 [英] pandas df.to_parquet write to multiple smaller files

问题描述

是否可以使用 Pandas 的 DataFrame.to_parquet 功能将写入拆分为多个近似所需大小的文件?

Is it possible to use Pandas' DataFrame.to_parquet functionality to split writing into multiple files of some approximate desired size?

我有一个非常大的 DataFrame (100M x 100)，并且正在使用 df.to_parquet('data.snappy', engine='pyarrow', compression='snappy') 写入一个文件，但这会生成一个大约 4GB 的文件.相反，我希望将其拆分为许多 ~100MB 的文件.

I have a very large DataFrame (100M x 100), and am using df.to_parquet('data.snappy', engine='pyarrow', compression='snappy') to write to a file, but this results in a file that's about 4GB. I'd instead like this split into many ~100MB files.

推荐答案

另一种选择是使用 pyarrow.parquet.write_to_dataset() 中的 partition_cols 选项:

One other option is to use the partition_cols option in pyarrow.parquet.write_to_dataset():

import pyarrow.parquet as pq
import numpy as np

# df is your dataframe
n_partition = 100
df["partition_idx"] = np.random.choice(range(n_partition), size=df.shape[0])
table = pq.Table.from_pandas(df, preserve_index=False)
pq.write_to_dataset(table, root_path="{path to dir}/", partition_cols=["partition_idx"])

这篇关于pandas df.to_parquet 写入多个较小的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！