Spark DataFrames之间的计算差异 [英] Computing difference between Spark DataFrames

问题描述

我有两个DataFrames df1和df2.我想计算第三个DataFrame ``df3，这样df3 = (df1 - df2)，即df1中存在的所有元素，而不是df2中存在的所有元素.是否有内置库函数来实现类似df1.subtract(df2)的功能?

I have two DataFrames df1 and df2. I want to compute a third DataFrame ``df3 such that df3 = (df1 - df2) i.e all elements present in df1 but not in df2. Is there any in-built library function to achieve that something like df1.subtract(df2)?

推荐答案

您可能正在搜索except函数:

You are probably searching for except function: http://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame

根据说明:

def除了(其他:DataFrame):DataFrame

def except(other: DataFrame): DataFrame

返回一个新的DataFrame，此框架中包含行，但不包含行 另一个框架.这等效于SQL中的EXCEPT.

Returns a new DataFrame containing rows in this frame but not in another frame. This is equivalent to EXCEPT in SQL.

这篇关于Spark DataFrames之间的计算差异的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！