如果某一列是另一列的成员，如何过滤Spark数据帧 [英] How to filter Spark dataframe if one column is a member of another column

问题描述

我有一个包含两列(一个字符串和一个字符串数组)的数据框:

I have a dataframe with two columns(one string and one array of string):

root
 |-- user: string (nullable = true)
 |-- users: array (nullable = true)
 |    |-- element: string (containsNull = true)

如何过滤数据框，以便结果数据框仅包含user位于users中的行?

How can I filter the dataframe so that the result dataframe only contains rows that user is in users?

推荐答案

当然，这是可能的，但并不难.为此，您可以使用UDF.

Sure, It's possible and not so hard. To achieve this you may use a UDF.

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

val df = sc.parallelize(Array(
  ("1", Array("1", "2", "3")),
  ("2", Array("1", "2", "2", "3")),
  ("3", Array("1", "2"))
)).toDF("user", "users")

val inArray = udf((id: String, array: scala.collection.mutable.WrappedArray[String]) => array.contains(id), BooleanType)

df.where(inArray($"user", $"users")).show()

输出为:

+----+------------+
|user|       users|
+----+------------+
|   1|   [1, 2, 3]|
|   2|[1, 2, 2, 3]|
+----+------------+

这篇关于如果某一列是另一列的成员，如何过滤Spark数据帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！