重命名书面CSV文件Spark [英] Rename written CSV file Spark

问题描述

我正在运行spark 2.1，我想将带有结果的csv写入Amazon S3. 重新分区后，csv文件具有一个较长的kryptic名称，我想将其更改为特定的文件名.

I'm running spark 2.1 and I want to write a csv with results into Amazon S3. After repartitioning the csv file has kind of a long kryptic name and I want to change that into a specific filename.

我正在使用databricks库写入S3.

I'm using the databricks lib for writing into S3.

dataframe
    .repartition(1)
    .write
    .format("com.databricks.spark.csv")
    .option("header", "true")
    .save("folder/dataframe/")

是否可以在以后重命名文件，甚至直接使用正确的名称保存文件?我已经在寻找解决方案了，还没有发现太多.

Is there a way to rename the file afterwards or even save it directly with the correct name? I've already looked for solutions and havent found much.

谢谢

推荐答案

您可以在下面使用它重命名输出文件.

You can use below to rename the output file.

dataframe.repartition(1).write.format("com.databricks.spark.csv").option("header", "true").save("folder/dataframe/")

import org.apache.hadoop.fs._

val fs = FileSystem.get(sc.hadoopConfiguration)

val filePath = "folder/dataframe/"
val fileName = fs.globStatus(new Path(filePath+"part*"))(0).getPath.getName

fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))

这篇关于重命名书面CSV文件Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！