读取pySpark中的文件范围 [英] Read range of files in pySpark

问题描述

我需要读取pySpark中的连续文件.以下对我有用.

I need to read contiguous files in pySpark. The following works for me.

from pyspark.sql import SQLContext    
file = "events.parquet/exportDay=2015090[1-7]"
df = sqlContext.read.load(file)

我如何读取文件8-14?

How do I read files 8-14?

推荐答案

使用花括号.

file ="events.parquet/exportDay = 201509 {08,09,10,11,12,13,14}"

file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"

这是关于堆栈溢出的类似问题: Pyspark使用正则表达式glob 选择文件子集.他们建议要么使用花括号，要么执行多次读取然后合并对象(无论是RDD还是数据帧，还是应该有某种方式).

Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).

这篇关于读取pySpark中的文件范围的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！