在 Spark 中将多个小文件合并为几个大文件 [英] merge multiple small files in to few larger files in Spark

问题描述

我通过 Spark 使用 hive.我的 spark 代码中有一个 Insert into partitioned table 查询.输入数据为 200+gb.当 Spark 写入分区表时，它会吐出非常小的文件(以 kb 为单位的文件).所以现在输出分区表文件夹有 5000 多个小 kb 文件.我想将这些合并到几个大 MB 文件中，可能只有几个 200mb 的文件.我厌倦了使用 hive 合并设置，但它们似乎不起作用.

I using hive through Spark. I have a Insert into partitioned table query in my spark code. The input data is in 200+gb. When Spark is writing to a partitioned table, it is spitting very small files(files in kb's). so now the output partitioned table folder have 5000+ small kb files. I want to merge these in to few large MB files, may be about few 200mb files. I tired using hive merge settings, but they don't seem to work.

'val result7A = hiveContext.sql("set hive.exec.dynamic.partition=true")

 val result7B = hiveContext.sql("set hive.exec.dynamic.partition.mode=nonstrict")

val result7C = hiveContext.sql("SET hive.merge.size.per.task=256000000")

val result7D = hiveContext.sql("SET hive.merge.mapfiles=true")

val result7E = hiveContext.sql("SET hive.merge.mapredfiles=true")

val result7F = hiveContext.sql("SET hive.merge.sparkfiles = true")

val result7G = hiveContext.sql("set hive.aux.jars.path=c:\\Applications\\json-serde-1.1.9.3-SNAPSHOT-jar-with-dependencies.jar")

val result8 = hiveContext.sql("INSERT INTO TABLE partition_table PARTITION (date) select a,b,c from partition_json_table")'

上述 hive 设置在 mapreduce hive 执行中工作，并输出指定大小的文件.是否有任何选项可以执行此 Spark 或 Scala?

The above hive settings work in a mapreduce hive execution and spits out files of specified size. Is there any option to do this Spark or Scala?

推荐答案

您可能想尝试使用 DataFrame.coalesce 方法；它返回一个具有指定分区数的 DataFrame(每个分区在插入时成为一个文件).因此，使用您插入的记录数和每条记录的典型大小，您可以估计如果您想要大约 200MB 的文件，要合并多少个分区.

You may want to try using the DataFrame.coalesce method; it returns a DataFrame with the specified number of partitions (each of which becomes a file on insertion). So using the number of records you are inserting and the typical size of each record, you can estimate how many partitions to coalesce to if you want files of ~200MB.

这篇关于在 Spark 中将多个小文件合并为几个大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！