基于公共值的 Spark 过滤器 DataFrames [英] Spark filter DataFrames based on common values

问题描述

我有 DF1 和 DF2.第一个有一列new_id"，第二个有一列db_id"

I have DF1 and DF2. First one has a column "new_id", the second has a column "db_id"

我需要过滤掉第一个 DataFrame 中的所有行，其中 new_id 的值不在 db_id 中.

I need to FILTER OUT all the rows from the first DataFrame, where the value of new_id is not in db_id.

val new_id = Seq(1, 2, 3, 4)
val db_id = Seq(1, 4, 5, 6, 10)

然后我需要 new_id == 1 和 4 的行留在 df1 中并删除 news_id = 2 和 3 的行，因为 2 和 3 不在 db_id 中

Then I need the rows with new_id == 1 and 4 to stay in df1 and delete the rows with news_id = 2 and 3 since 2 and 3 are not in db_id

这里有很多关于 DataFrame 的问题.我可能错过了这个.对不起，如果这是重复的.

There is a ton of questions on DataFrames here. I might have missed this one. Sorry if that is a duplicate.

p.s 如果重要的话，我正在使用 Scala.

p.s I am using Scala if that matters.

推荐答案

你需要的是一个左半角:

What you need is an left-semi jon:

import spark.implicits._

val DF1 = Seq(1,3).toDF("new_id")
val DF2 = Seq(1,2).toDF("db_id")


DF1.as("df1").join(DF2.as("df2"),$"df1.new_id"===$"df2.db_id","leftsemi")
.show()

+------+
|new_id|
+------+
|     1|
+------+

这篇关于基于公共值的 Spark 过滤器 DataFrames的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！