为什么我的 BroadcastHashJoin 比 Spark 中的 ShuffledHashJoin 慢 [英] Why my BroadcastHashJoin is slower than ShuffledHashJoin in Spark

问题描述

我在 Spark 中使用 javaHiveContext 执行连接.

I execute a join using a javaHiveContext in Spark.

大表是 1,76Gb，有 1 亿条记录.

The big table is 1,76Gb and has 100 millions record.

第二个表是 273Mb，有 1000 万条记录.

The second table is 273Mb and has 10 millions record.

我得到一个 JavaSchemaRDD 并且我在它上面调用 count() :

I get a JavaSchemaRDD and I call count() on it:

String query="select attribute7,count(*) from ft,dt where ft.chiavedt=dt.chiavedt group by attribute7";

JavaSchemaRDD rdd=sqlContext.sql(query);

System.out.println("count="+rdd.count());

如果我强制使用 broadcastHashJoin (SET spark.sql.autoBroadcastJoinThreshold=290000000) 并在 5 个具有 8 核和 20Gb 内存的节点上使用 5 个执行程序，它将在 100 秒内执行.如果我不强制广播，它会在 30 秒内执行.

If I force a broadcastHashJoin (SET spark.sql.autoBroadcastJoinThreshold=290000000) and use 5 executor on 5 node with 8 core and 20Gb of memory it is executed in 100 sec. If i don't force broadcast it is executed in 30 sec.

注意表存储为 Parquet 文件.

N.B. the tables are stored as Parquet file.

推荐答案

问题的根源很可能是广播成本.为了简单起见，假设您在较大的 RDD 中有 1800MB，在较小的 RDD 中有 300MB.假设有 5 个执行程序并且之前没有分区所有数据的五分之一应该已经在正确的机器上.在标准连接的情况下，它会留下约 1700MB 用于 shuffle.

Most likely the source of the problem is a cost of broadcasting. To make things simple lets assume that you have 1800MB in the larger RDD and 300MB in the smaller one. Assuming 5 executors and no previous partitioning a fifth of all data should be already on the correct machine. It lefts ~1700MB for shuffling in case of standard join.

对于广播加入，较小的 RDD 必须转移到所有节点.这意味着要传输大约 1500MB 的数据.如果您添加所需的与驱动程序的通信，则意味着您必须以更昂贵的方式移动相当数量的数据.必须首先收集广播数据，然后才能将其转发给所有工作人员.

For broadcast join the smaller RDD has to be transfered to all nodes. It means around 1500MB data to be transfered. If you add required communication with driver it means you have to move a comparable amount of data in a much more expensive way. A broadcasted data has to be collected first and only after that can be forwarded to all the workers.

这篇关于为什么我的 BroadcastHashJoin 比 Spark 中的 ShuffledHashJoin 慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！