共划分火花SQL连接 [英] Co-partitioned joins in spark SQL

问题描述

是否有提供联合分区连接星火SQL数据源中的任何实现的 - 最有可能通过CoGroupRDD？我没看到现有的星火codeBase类中的任何用途。

Are there any implementations of Spark SQL DataSources that offer Co-partition joins - most likely via the CoGroupRDD? I did not see any uses within the existing Spark codebase.

的动机将大大减少在这两个表中有相同数量和分区键的范围相同的情况下，洗牌交通：在这种情况下会有一个 MX1 ，而不是<强> M×N个洗牌扇出。

The motivation would be to greatly reduce the shuffle traffic in the case that two tables have the same number and same ranges of partitioning keys: in that case there would be a Mx1 instead of an MxN shuffle fanout.

唯一的大规模实施连接presently在星火SQL似乎 ShuffledHashJoin 的 - 其中确实的需要的M×N个洗牌扇出并且因此是昂贵的。

The only large-scale implementation of joins presently in Spark SQL seems to be ShuffledHashJoin - which does require the MxN shuffle fanout and thus is expensive.

推荐答案

我认为你正在寻找的斗加入应在星火2.0现身优化。

在1.6你可以完成类似的东西，但只有通过缓存中的数据。 SPARK-4849

In 1.6 you can accomplish something similar, but only by caching the data. SPARK-4849

这篇关于共划分火花SQL连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！