为什么在 SQL 查询中使用 UDF 会导致笛卡尔积? [英] Why using a UDF in a SQL query leads to cartesian product?
问题描述
我看到了 Databricks-问题和不明白
- 为什么使用 UDF 会导致笛卡尔积而不是完整的外连接?显然，笛卡尔积会比完整的外部联接多得多(Joins 是一个例子)这是一个潜在的表现命中.
- 在Databricks-问题?
- Why using UDFs leads to a Cartesian product instead of a full outer join? Obviously the Cartesian product would be a lot more rows than a full outer join(Joins is an example) which is a potential performance hit.
- Any way to force an outer join over the Cartesian product in the example given in Databricks-Question?
引用 Databricks-问题在这里:
我有一个使用 SQLContext 执行的 Spark Streaming 应用程序关于流数据的 SQL 语句.当我在Scala，流应用的性能下降显着地.详情如下:
I have a Spark Streaming application that uses SQLContext to execute SQL statements on streaming data. When I register a custom UDF in Scala, the performance of the streaming application degrades significantly. Details below:
声明 1:
select col1, col2 from table1 as t1 join table2 as t2 on t1.foo = t2.bar
声明 2:
select col1, col2 from table1 as t1 join table2 as t2 on equals(t1.foo,t2.bar)
我使用 SQLContext 注册自定义 UDF，如下所示:
I register a custom UDF using SQLContext as follows:
sqlc.udf.register("equals", (s1: String, s2:String) => s1 == s2)
在相同的输入和 Spark 配置上，Statement2 性能与 Statement1 相比明显更差(接近 100 倍).
On the same input and Spark configuration, Statement2 performance significantly worse(close to 100X) compared to Statement1.
推荐答案
为什么使用 UDF 会导致笛卡尔积而不是全外连接?
Why using UDFs leads to a Cartesian product instead of a full outer join?
使用 UDF 需要笛卡尔积的原因很简单.由于您传递了一个可能具有无限域和非确定性行为的任意函数，因此确定其值的唯一方法是传递参数和求值.这意味着您只需检查所有可能的对.
The reason why using UDFs require Cartesian product is quite simple. Since you pass an arbitrary function with possibly infinite domain and non-deterministic behavior the only way to determine its value is to pass arguments and evaluate. It means you simply have to check all possible pairs.
另一方面，简单的相等具有可预测的行为.如果您使用 t1.foo = t2.bar 条件，您可以简单地将 t1 和 t2 行按 foo 和bar 分别得到预期的结果.
Simple equality from the other hand has a predictable behavior. If you use t1.foo = t2.bar condition you can simply shuffle t1 and t2 rows by foo and bar respectively to get expected result.
准确地说，在关系代数中，外连接实际上是使用自然连接来表达的.除此之外的任何事情都只是一种优化.
And just to be precise in the relational algebra outer join is actually expressed using natural join. Anything beyond that is simply an optimization.
任何在笛卡尔积上强制外连接的方法
Any way to force an outer join over the Cartesian product
不是真的，除非你想修改 Spark SQL 引擎.
Not really, unless you want to modify Spark SQL engine.
这篇关于为什么在 SQL 查询中使用 UDF 会导致笛卡尔积?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
