在Spark SQL中连接两个DataFrame并选择仅一个的列 [英] Joining two DataFrames in Spark SQL and selecting columns of only one

问题描述

我在Spark SQL中有两个DataFrame( D1和D2 ).

I have two DataFrames in Spark SQL (D1 and D2).

我正在尝试内部加入他们两个D1.join(D2, "some column") 并只获取​​仅D1的数据，而不是完整的数据集.

I am trying to inner join both of them D1.join(D2, "some column") and get back data of only D1, not the complete data set.

D1和D2都具有相同的列.

Both D1 and D2 are having the same columns.

有人可以帮我吗?

我正在使用Spark 1.6.

I am using Spark 1.6.

推荐答案

假设您要加入"id"列.然后，您可以编写:

Let say you want to join on "id" column. Then you could write :

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._    
d1.as("d1").join(d2.as("d2"), $"d1.id" === $"d2.id").select($"d1.*")

这篇关于在Spark SQL中连接两个DataFrame并选择仅一个的列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！