在Spark中同时读取几个实木复合地板文件 [英] Read few parquet files at the same time in Spark

问题描述

我可以使用*(星号)同时读取几个json文件:

I can read few json-files at the same time using * (star):

sqlContext.jsonFile('/path/to/dir/*.json')

有什么方法可以对镶木地板做同样的事情?星空无效.

Is there any way to do the same thing for parquet? Star doesn't works.

推荐答案

请参见此问题在火花jira上.从1.4开始支持.

See this issue on the spark jira. It is supported from 1.4 onwards.

如果不升级到1.4，则可以指向顶层目录:

Without upgrading to 1.4, you could either point at the top level directory:

sqlContext.parquetFile('/path/to/dir/')

这将加载目录中的所有文件.另外，您可以使用HDFS API查找所需的文件，然后将它们传递给parquetFile(它接受varargs).

which will load all files in the directory. Alternatively, you could use the HDFS API to find the files you want, and pass them to parquetFile (it accepts varargs).

这篇关于在Spark中同时读取几个实木复合地板文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！