Spark中的Round Robin分区如何工作? [英] How does Round Robin partitioning in Spark work?
问题描述
我很难理解Spark中的Round Robin分区.考虑以下示例.我将大小为3的Seq分为3个分区:
I've trouble to understand Round Robin Partitioning in Spark. Consider the following exampl. I split a Seq of size 3 into 3 partitions:
val df = Seq(0,1,2).toDF().repartition(3)
df.explain
== Physical Plan ==
Exchange RoundRobinPartitioning(3)
+- LocalTableScan [value#42]
现在,如果我检查分区,我将得到:
Now if I inspect the partitions, I get:
df
.rdd
.mapPartitionsWithIndex{case (i,rows) => Iterator((i,rows.size))}
.toDF("partition_index","number_of_records")
.show
+---------------+-----------------+
|partition_index|number_of_records|
+---------------+-----------------+
| 0| 0|
| 1| 2|
| 2| 1|
+---------------+-----------------+
如果我对大小为8的Seq进行同样的操作并将其分成8个分区,则歪斜会更严重:
If I do the same with Seq of size 8 and split it into 8 partitions, I get even worse skew:
(0 to 7).toDF().repartition(8)
.rdd
.mapPartitionsWithIndex{case (i,rows) => Iterator((i,rows.size))}
.toDF("partition_index","number_of_records")
.show
+---------------+-----------------+
|partition_index|number_of_records|
+---------------+-----------------+
| 0| 0|
| 1| 0|
| 2| 0|
| 3| 0|
| 4| 0|
| 5| 0|
| 6| 4|
| 7| 4|
+---------------+-----------------+
有人可以解释这种行为.据我了解循环分区,所有分区的显示大小均相同.
Can somebody explain this behavior. As far as I understand round robin partitioning, all partitions show be ~same size.
推荐答案
(已检查Spark版本2.1-2.4)
As far as I can see from ShuffleExchangeExec
code, Spark tries to partition the rows directly from original partitions (via mapPartitions
) without bringing anything to the driver.
逻辑是从随机选择的目标分区开始,然后以循环方法将分区分配给行.请注意,为每个源分区都选择了开始"分区,并且可能会发生冲突.
The logic is to start with a randomly picked target partition and then assign partitions to the rows in a round-robin method. Note that "start" partition is picked for each source partition and there could be collisions.
最终分布取决于许多因素:源/目标分区的数量以及数据框中的行数.
The final distribution depends on many factors: a number of source/target partitions and the number of rows in your dataframe.
这篇关于Spark中的Round Robin分区如何工作?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!