使用 Python 编写 Parquet 文件的方法? [英] Methods for writing Parquet files using Python?

问题描述

我无法找到允许使用 Python 编写 Parquet 文件的库.如果我可以结合使用 Snappy 或类似的压缩机制，我会加分.

I'm having trouble finding a library that allows Parquet files to be written using Python. Bonus points if I can use Snappy or a similar compression mechanism in conjunction with it.

到目前为止，我发现的唯一方法是使用带有 pyspark.sql.DataFrame Parquet 支持的 Spark.

Thus far the only method I have found is using Spark with the pyspark.sql.DataFrame Parquet support.

我有一些脚本需要编写非 Spark 作业的 Parquet 文件.是否有任何方法可以在 Python 中编写不涉及 pyspark.sql 的 Parquet 文件?

I have some scripts that need to write Parquet files that are not Spark jobs. Is there any approach to writing Parquet files in Python that doesn't involve pyspark.sql?

推荐答案

更新(2017 年 3 月):目前有 2 个能够编写的库em> Parquet 文件:

Update (March 2017): There are currently 2 libraries capable of writing Parquet files:

1. fastparquet
2. pyarrow

它们似乎仍在大力开发中，并且带有许多免责声明(例如不支持嵌套数据)，因此您必须检查它们是否支持您需要的一切.

Both of them are still under heavy development it seems and they come with a number of disclaimers (no support for nested data e.g.), so you will have to check whether they support everything you need.

旧答案:

截至 2016 年，似乎没有能够编写 Parquet 文件的纯 python 库.

As of 2.2016 there seems to be NO python-only library capable of writing Parquet files.

如果您只需要阅读 Parquet 文件，则有 蟒蛇实木复合地板.

If you only need to read Parquet files there is python-parquet.

作为一种解决方法，您将不得不依赖其他一些过程，例如pyspark.sql(它使用 Py4J 并在 JVM 上运行，因此不能直接从您的普通 CPython 程序中使用).

As a workaround you will have to rely on some other process like e.g. pyspark.sql (which uses Py4J and runs on the JVM and can thus not be used directly from your average CPython program).

这篇关于使用 Python 编写 Parquet 文件的方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！