在Spark中，如果数据帧中没有行，如何在文件中写入标头? [英] In Spark, how to write header in a file, if there is no row in a dataframe?

问题描述

如果数据帧中没有行，我想在文件中写入标头.目前，当我向文件写入空数据帧时，会创建文件，但其中没有标头.

I want to write a header in a file if there is no row in dataframe, Currently when I write an empty dataframe to a file then file is created but it does not have header in it.

I am writing dataframe using these setting and command:
Dataframe.repartition(1) \
         .write \
         .format("com.databricks.spark.csv") \
         .option("ignoreLeadingWhiteSpace", False) \
         .option("ignoreTrailingWhiteSpace", False) \
         .option("header", "true") \
         .save('/mnt/Bilal/Dataframe');

即使数据帧中没有数据行，我也希望文件中的标题行.

I want the header row in the file, even if there is no data row in a dataframe.

推荐答案

如果您只想拥有头文件.您可以使用向左折以创建带有空白的每一列，并将其另存为csv.我没有使用pyspark，但是这可以在scala中完成.大多数代码都应该可重用，您只需将其转换为pyspark

if you want to have just header file. you can use fold left to create each column with white space and save that as your csv. I have not used pyspark but this is how it can be done in scala. majority of the code should be reusable you will have to just work on converting it to pyspark

val path ="/user/test"
val newdf=df.columns.foldleft(df){(tempdf,cols)=>
tempdf.withColumn(cols, lit(""))}

创建用于写入头文件的方法

create a method for writing the header file

 def createHeaderFile(headerFilePath: String, colNames: Array[String]) {

//format header file path
val fileName = "yourfileName.csv"
val headerFileFullName = "%s/%s".format(headerFilePath, fileName)

    val hadoopConfig = new Configuration()
val fileSystem = FileSystem.get(hadoopConfig)
val output = fileSystem.create(new Path(headerFileFullName))
val writer = new PrintWriter(output)

for (h <- colNames) {
  writer.write(h + ",")
}
writer.write("\n")
writer.close()
}

在您的DF上调用

 createHeaderFile(path, newdf.columns)

这篇关于在Spark中，如果数据帧中没有行，如何在文件中写入标头?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！