如何根据行数重新分区 Spark 数据帧? [英] How to repartition Spark dataframe depending on row count?

问题描述

我写了一个简单的程序来请求一个巨大的数据库.为了导出我的结果，我写了这个函数:

I wrote a simple program that request a huge database. To export my result, I wrote this function:

result.coalesce(1).write.options(Map("header" -> "true", "delimiter"  > ";")).csv(mycsv.csv)

我使用 coalesce 方法只得到一个文件作为输出.问题是结果文件包含超过一百万行.所以，我无法在 Excel 中打开它...

I use the coalesce method to have only get one file as an output. The problem is that the result file contains more than one million lines. So, I couldn't open it in Excel...

因此，我考虑使用一种方法(或使用 for 循环编写自己的函数)，该方法可以创建与我的文件中的行数相关的分区.但我不知道我该怎么做.

So, I thought about using a method (or write my own function using a for loop) that can create partitions related to the number of the lines in my file. But I have no idea how can I do this.

我的想法是，如果我的行少于一百万，我将有一个分区.如果我有超过一百万 => 两个分区，2 百万 => 3 个分区等等.

My idea is that if I have less than one million line, I will have one partition. If I have more than one million => two partitions, 2 millions => 3 partitions and so on.

可以做这样的事情吗?

推荐答案

您可以根据数据框中的行数更改分区数.

You can change the number of partition depending on the number of rows in the dataframe.

例如:

val rowsPerPartition = 1000000
val partitions = (1 + df.count() / rowsPerPartition).toInt

val df2 = df.repartition(numPartitions=partitions)

然后像以前一样将新数据帧写入 csv 文件.

Then write the new dataframe to a csv file as before.

注意:可能需要使用 repartition 而不是 coalesce 来确保每个分区中的行数大致相等，请参阅 Spark - repartition() 与合并 ().

Note: it may be required to use repartition instead of coalesce to make sure the number of rows in each partition are roughly equal, see Spark - repartition() vs coalesce().

这篇关于如何根据行数重新分区 Spark 数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！