写入时如何控制Spark作业创建的输出零件文件的数量? [英] How to control the number of output part files created by Spark job upon writing?

问题描述

我有几个Spark作业，每天处理数千个文件.文件大小可能从MB到GB.完成工作后，我通常使用以下代码保存

Hi I am having couple of Spark jobs which processes thousands of files every day. File size may very from MBs to GBs. After finishing job I usually save using the following code

finalJavaRDD.saveAsParquetFile("/path/in/hdfs"); OR
dataFrame.write.format("orc").save("/path/in/hdfs") //storing as ORC file as of Spark 1.4

Spark作业会在最终输出目录中创建大量小零件文件.据我了解，Spark为每个分区/任务创建零件文件，如果我错了，请纠正我.我们如何控制Spark创建的零件文件的数量?最后，我想使用这些parquet/orc目录创建Hive表，并且听说在没有大量小文件的情况下Hive运行缓慢.请指导我是Spark的新手.预先感谢.

Spark job creates plenty of small part files in final output directory. As far as I understand Spark creates part file for each partition/task please correct me if I am wrong. How do we control amount of part files Spark creates? Finally I would like to create Hive table using these parquet/orc directory and I heard Hive is slow when we have large no of small files. Please guide I am new to Spark. Thanks in advance.

推荐答案

您可能想尝试使用

You may want to try using the DataFrame.coalesce method to decrease the number of partitions; it returns a DataFrame with the specified number of partitions (each of which becomes a file on insertion).

要增加或减少分区，可以使用Dataframe.repartition功能. 但是coalesce不会引起随机播放，而repartition却可以引起随机播放.

To increase or decrease the partitions you can use Dataframe.repartition function. But coalesce does not cause shuffle while repartition does.

这篇关于写入时如何控制Spark作业创建的输出零件文件的数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！