在 Spark Streaming 中更改输出文件名 [英] Change output file name in Spark Streaming

问题描述

我正在运行一个 Spark 作业，就逻辑而言，它的性能非常好.但是，当我使用 saveAsTextFile 将文件保存在 s3 存储桶中时，我的输出文件的名称采用 part-00000、part-00001 等格式.有没有办法改变输出文件名?

谢谢.

解决方案

在 Spark 中，您可以使用 saveAsNewAPIHadoopFile 并将 hadoop 配置中的 ma​​preduce.output.basename 参数设置为更改前缀(只是部分"前缀)

val hadoopConf = new Configuration()hadoopConf.set("mapreduce.output.basename", "yourPrefix")yourRDD.map(str => (null, str)).saveAsNewAPIHadoopFile(s"$outputPath/$dirName", classOf[NullWritable], classOf[String],classOf[TextOutputFormat[NullWritable, String]], hadoopConf)

您的文件将命名为:yourPrefix-r-00001

在 hadoop 和 Spark 中，您可以在输出中有多个文件，因为您可以有多个 reducer(hadoop)或多个分区(spark).然后您需要保证每个文件的名称都是唯一的，这就是为什么不能覆盖文件名最后一部分的序列号.

但如果您想更好地控制文件名，您可以扩展 TextOutputFormat 或 FileOutputFormat 并覆盖 getUniqueFile 方法.>

I am running a Spark job which performs extremely well as far as the logic goes. However, the name of my output files are in the format part-00000,part-00001 etc., when I use saveAsTextFile to save the files in a s3 bucket. Is there a way to change the output filename?

Thank you.

解决方案

In Spark, you can use saveAsNewAPIHadoopFile and set mapreduce.output.basename parameter in hadoop configuration to change the prefix (Just the "part" prefix)

val hadoopConf = new Configuration()
hadoopConf.set("mapreduce.output.basename", "yourPrefix")

yourRDD.map(str => (null, str))
        .saveAsNewAPIHadoopFile(s"$outputPath/$dirName", classOf[NullWritable], classOf[String],
          classOf[TextOutputFormat[NullWritable, String]], hadoopConf)

Your files will be named like: yourPrefix-r-00001

In hadoop and Spark, you can have more than one file in the output since you can have more than one reducer(hadoop) or more than one partition(spark). Then you need to warranty unique names for each of them, that is why it is not possible to override the sequence number at the last part of the filename.

But if you want to have more control of your filename, you can extend TextOutputFormat or FileOutputFormat and override the getUniqueFile method.

这篇关于在 Spark Streaming 中更改输出文件名的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！