重命名写入的 CSV 文件 Spark [英] Rename written CSV file Spark

问题描述

我正在运行 spark 2.1，并且我想将一个带有结果的 csv 写入 Amazon S3.重新分区后，csv 文件的名称有点长，我想将其更改为特定的文件名.

I'm running spark 2.1 and I want to write a csv with results into Amazon S3. After repartitioning the csv file has kind of a long kryptic name and I want to change that into a specific filename.

我正在使用 databricks 库来写入 S3.

I'm using the databricks lib for writing into S3.

dataframe
    .repartition(1)
    .write
    .format("com.databricks.spark.csv")
    .option("header", "true")
    .save("folder/dataframe/")

有没有办法事后重命名文件，甚至直接用正确的名称保存?我已经在寻找解决方案，但没有找到太多.

Is there a way to rename the file afterwards or even save it directly with the correct name? I've already looked for solutions and havent found much.

谢谢

推荐答案

您可以使用下面的命令重命名输出文件.

You can use below to rename the output file.

dataframe.repartition(1).write.format("com.databricks.spark.csv").option("header", "true").save("folder/dataframe/")

import org.apache.hadoop.fs._

val fs = FileSystem.get(sc.hadoopConfiguration)

val filePath = "folder/dataframe/"
val fileName = fs.globStatus(new Path(filePath+"part*"))(0).getPath.getName

fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))

这篇关于重命名写入的 CSV 文件 Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！