两个分区数据帧之间的 Spark 共置连接 [英] Spark colocated join between two partitioned dataframes

问题描述

对于 Spark 1.6.0 中两个 DataFrames 之间的以下连接

For the following join between two DataFrames in Spark 1.6.0

val df0Rep = df0.repartition(32, col("a")).cache
val df1Rep = df1.repartition(32, col("a")).cache
val dfJoin = df0Rep.join(df1Rep, "a")
println(dfJoin.count)

这是否不仅共同分区而且共同定位?我知道对于 RDD，如果使用相同的分区器并在相同的操作中混洗，则连接将位于同一位置.但是数据帧呢?谢谢.

Does this join not only co-partitioned but also co-located? I know that for RDDs if using the same partitioner and shuffled in the same operation, the join would be co-located. But what about dataframes? Thank you.

推荐答案

[https://medium.com/@achilleus/https-medium-com-joins-in-apache-spark-part-3-1d40c1e51e1c]

根据上面提供的文章链接Sort-Merge join是默认的join，想补充一点

According to the article link provided above Sort-Merge join is the default join, would like to add important point

为了 Sort-Merge join 的理想性能，重要的是所有具有相同连接键值的行在相同的划分.这保证了臭名昭著的分区交换(shuffle)执行者之间.并置分区可以避免不必要的数据洗牌.数据需要在连接键中均匀分布.这连接键的数量足够独特，因此它们可以相等分布在集群中以实现最大并行度可用分区

For Ideal performance of Sort-Merge join, it is important that all rows having the same value for the join key are available in the same partition. This warrants for the infamous partition exchange(shuffle) between executors. Collocated partitions can avoid unnecessary data shuffle. Data needs to be evenly distributed n the join keys. The number of join keys is unique enough so that they can be equally distributed across the cluster to achieve the max parallelism from the available partitions

这篇关于两个分区数据帧之间的 Spark 共置连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！