在Spark中，如果数据帧中没有行，如何在文件中写入标题? [英] In Spark, how to write header in a file, if there is no row in a dataframe?

问题描述

如果数据框中没有行，我想在文件中写入标题，目前当我将空数据框写入文件时，会创建文件，但其中没有标题.

I want to write a header in a file if there is no row in dataframe, Currently when I write an empty dataframe to a file then file is created but it does not have header in it.

I am writing dataframe using these setting and command:
Dataframe.repartition(1) \
         .write \
         .format("com.databricks.spark.csv") \
         .option("ignoreLeadingWhiteSpace", False) \
         .option("ignoreTrailingWhiteSpace", False) \
         .option("header", "true") \
         .save('/mnt/Bilal/Dataframe');

我想要文件中的标题行，即使数据框中没有数据行.

I want the header row in the file, even if there is no data row in a dataframe.

推荐答案

如果你只想拥有头文件.您可以使用 fold left 创建带有空格的每一列，并将其保存为您的 csv.我没有使用过 pyspark，但这就是在 Scala 中可以做到的.大多数代码应该是可重用的，您只需将其转换为 pyspark

if you want to have just header file. you can use fold left to create each column with white space and save that as your csv. I have not used pyspark but this is how it can be done in scala. majority of the code should be reusable you will have to just work on converting it to pyspark

val path ="/user/test"
val newdf=df.columns.foldleft(df){(tempdf,cols)=>
tempdf.withColumn(cols, lit(""))}

创建写入头文件的方法

 def createHeaderFile(headerFilePath: String, colNames: Array[String]) {

//format header file path
val fileName = "yourfileName.csv"
val headerFileFullName = "%s/%s".format(headerFilePath, fileName)

    val hadoopConfig = new Configuration()
val fileSystem = FileSystem.get(hadoopConfig)
val output = fileSystem.create(new Path(headerFileFullName))
val writer = new PrintWriter(output)

for (h <- colNames) {
  writer.write(h + ",")
}
writer.write("\n")
writer.close()
}

在你的 DF 上调用它

call it on your DF

 createHeaderFile(path, newdf.columns)

这篇关于在Spark中，如果数据帧中没有行，如何在文件中写入标题?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！