为什么Spark的重新分区没有将数据平衡到分区中? [英] Why the Spark's repartition didn't balance data into partitions?

问题描述

>>> rdd = sc.parallelize(range(10), 2)
>>> rdd.glom().collect()
[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
>>> rdd.repartition(3).glom().collect()
[[], [0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
>>>

第一个分区为空?为什么?非常感谢您告诉我原因.

The first partition is empty? Why? I really appreciate you telling me the reasons.

推荐答案

之所以会发生这种情况，是因为Spark不会对单个元素进行混洗，而是对数据块进行混洗-

That happens because Spark doesn't shuffle individual elements but rather blocks of data - with minimum batch size equal to 10.

因此，如果您的元素少于每个分区的元素，Spark将不会分隔分区的内容.

So if you have less elements than that per partition, Spark won't separate content of partitions.

这篇关于为什么Spark的重新分区没有将数据平衡到分区中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！