Spark在两个分区数据帧之间共置联接 [英] Spark colocated join between two partitioned dataframes

问题描述

对于Spark 1.6.0中两个DataFrames之间的以下联接

For the following join between two DataFrames in Spark 1.6.0

val df0Rep = df0.repartition(32, col("a")).cache
val df1Rep = df1.repartition(32, col("a")).cache
val dfJoin = df0Rep.join(df1Rep, "a")
println(dfJoin.count)

此连接不仅共同分区，而且共同定位吗?我知道对于RDD，如果使用相同的分区程序并在相同的操作中进行混洗，则连接将位于同一位置.但是数据帧呢?谢谢.

Does this join not only co-partitioned but also co-located? I know that for RDDs if using the same partitioner and shuffled in the same operation, the join would be co-located. But what about dataframes? Thank you.

推荐答案

[根据上面提供的文章链接，排序-合并"联接是默认联接，想添加要点

According to the article link provided above Sort-Merge join is the default join, would like to add important point

为使Sort-Merge联接具有理想的性能，重要的是所有 具有相同连接键值的行可在同一行中使用 分割.这保证了臭名昭著的分区交换(随机播放) 执行者之间.并置的分区可以避免不必要的数据 洗牌.数据需要均匀地分布在连接键中.这 连接键的数量足够独特，因此它们可以相等 分布在整个群集中，以实现最大并行度 可用分区

For Ideal performance of Sort-Merge join, it is important that all rows having the same value for the join key are available in the same partition. This warrants for the infamous partition exchange(shuffle) between executors. Collocated partitions can avoid unnecessary data shuffle. Data needs to be evenly distributed n the join keys. The number of join keys is unique enough so that they can be equally distributed across the cluster to achieve the max parallelism from the available partitions

这篇关于Spark在两个分区数据帧之间共置联接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！