如果一列是另一列的成员，如何过滤 Spark 数据框 [英] How to filter Spark dataframe if one column is a member of another column

问题描述

我有一个包含两列的数据框(一个字符串和一个字符串数组):

I have a dataframe with two columns(one string and one array of string):

root
 |-- user: string (nullable = true)
 |-- users: array (nullable = true)
 |    |-- element: string (containsNull = true)

如何过滤数据框，以便结果数据框只包含 user 在 users 中的行?

How can I filter the dataframe so that the result dataframe only contains rows that user is in users?

推荐答案

当然，这是可能的，而且并不难.为此，您可以使用 UDF.

Sure, It's possible and not so hard. To achieve this you may use a UDF.

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

val df = sc.parallelize(Array(
  ("1", Array("1", "2", "3")),
  ("2", Array("1", "2", "2", "3")),
  ("3", Array("1", "2"))
)).toDF("user", "users")

val inArray = udf((id: String, array: scala.collection.mutable.WrappedArray[String]) => array.contains(id), BooleanType)

df.where(inArray($"user", $"users")).show()

输出为:

+----+------------+
|user|       users|
+----+------------+
|   1|   [1, 2, 3]|
|   2|[1, 2, 2, 3]|
+----+------------+

这篇关于如果一列是另一列的成员，如何过滤 Spark 数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！