如何使用 PySpark 读取目录下的 Parquet 文件? [英] How to read Parquet files under a directory using PySpark?

问题描述

我在网上搜索过，网上提供的解决方案没有解决我的问题.我正在尝试读取分层目录下的镶木地板文件.我收到以下错误.

I have searched online and the solutions provided online didn't resolve my issue. I am trying to read parquet files under a directory which are hierarchical. I am getting the following error.

'无法推断 Parquet 的架构.必须手动指定.;'

'Unable to infer schema for Parquet. It must be specified manually.;'

我的目录结构如下:dbfs:/mnt/sales/region/country/2020/08/04

My directory structure looks like: dbfs:/mnt/sales/region/country/2020/08/04

年文件夹下会有多个月份的子目录，而月份下的后续子目录则是几天.

There will be multiple sub-directories for months under the year folder and subsequent sub-directories under month for days.

我只想在销售级别阅读它们，这应该给我所有地区，我已经尝试了以下两个代码，但都没有奏效.请帮我解决这个问题.

I only want to read them at the sales level which should give me for all the regions and I've tried both of the below codes but neither of them worked. Please help me with this.

spark.read.parquet("dbfs:/mnt/sales/*")

或

spark.read.parquet("dbfs:/mnt/sales/")

推荐答案

你能试试这个选项吗?

df = spark.read.option("header","true").option("recursiveFileLookup","true").parquet("/path/to/root/")

这篇关于如何使用 PySpark 读取目录下的 Parquet 文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！