根据提供的列表过滤数组列 [英] Filter an array column based on a provided list

问题描述

我在数据框中具有以下类型:

I have the following types in a dataframe:

 root
 |-- id: string (nullable = true)
 |-- items: array (nullable = true)
 |    |-- element: string (containsNull = true)

输入:

val rawData = Seq(("id1",Array("item1","item2","item3","item4")),("id2",Array("item1","item2","item3")))
val data = spark.createDataFrame(rawData)

和项目列表:

 val filter_list = List("item1", "item2")

我想过滤掉filter_list中没有的项目，类似于array_contains的工作方式，但是它不适用于提供的字符串列表，只能使用单个值.

I would like to filter out items that are non in the filter_list, similar to how array_contains would function, but its not working on a provided list of strings, only a single value.

所以输出看起来像这样:

so the output would look like this:

val rawData = Seq(("id1",Array("item1","item2")),("id2",Array("item1","item2")))
val data = spark.createDataFrame(rawData)

我尝试使用以下UDF解决此问题，但我可能在Scala和Spark之间混合了类型:

I tried solving this with the following UDF, but I probably mix types between Scala and Spark:

def filterItems(flist: List[String]) = udf {
  (recs: List[String]) => recs.filter(item => flist.contains(item))
}

我正在使用Spark 2.2

I'm using Spark 2.2

谢谢！

推荐答案

您的代码几乎是正确的.您要做的就是将List替换为Seq

You code is almost right. All you have to do is replace List with Seq

def filterItems(flist: List[String]) = udf {
  (recs: Seq[String]) => recs.filter(item => flist.contains(item))
}

将签名从List[String] => UserDefinedFunction更改为SeqString] => UserDefinedFunction也很有意义，但这不是必需的.

It would also make sense to change signature from List[String] => UserDefinedFunction to SeqString] => UserDefinedFunction, but it is not required.

参考 SQL编程指南-数据类型.

这篇关于根据提供的列表过滤数组列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！