如何将pyspark数据帧写入HDFS，然后如何将其读回数据帧? [英] How to write pyspark dataframe to HDFS and then how to read it back into dataframe?

问题描述

我有一个非常大的 pyspark 数据框.所以我想对它的子集进行预处理，然后将它们存储到 hdfs.后来我想阅读所有这些并合并在一起.谢谢.

I have a very big pyspark dataframe. So I want to perform pre processing on subsets of it and then store them to hdfs. Later I want to read all of them and merge together. Thanks.

推荐答案

• 将 DataFrame 写入 HDFS (Spark 1.6).

  • writing DataFrame to HDFS (Spark 1.6).

    df.write.save('/target/path/', format='parquet', mode='append') ## df is an existing DataFrame object.
    

  一些格式选项是csv、parquet、json 等

  some of the format options are csv, parquet, json etc.

  • 从 HDFS (Spark 1.6) 读取数据帧.

  • reading DataFrame from HDFS (Spark 1.6).

  from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  sqlContext.read.format('parquet').load('/path/to/file') 
  

  format 方法接受parquet、csv、json 等参数

  the format method takes argument such as parquet, csv, json etc.

  这篇关于如何将pyspark数据帧写入HDFS，然后如何将其读回数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！