基于Apache Spark中数组中的单词过滤DataFrame [英] Filter DataFrame based on words in array in Apache Spark
问题描述
我试图通过仅获取包含数组中的单词的行来过滤数据集.我正在使用 contains 方法,它适用于字符串但不适用于数组.下面是代码
I am trying to Filter a Dataset by getting only those rows that contains words in array. I am using contains method,it works for string but not working for array. Below is code
val dataSet = spark.read.option("header","true").option("inferschema","true").json(path).na.drop.cache()
val threats_path = spark.read.textFile("src/main/resources/cyber_threats").collect()
val newData = dataSet.select("*").filter(col("_source.raw_text").contains(threats_path)).show()
它不起作用,因为威胁路径是字符串数组并包含字符串的工作.任何帮助将不胜感激.
It is not working becuase threats_path is array of strings and contains work for string. Any help would be appreciated.
推荐答案
您可以在列上使用 isin
udf
You can use isin
udf on columns
它会像,
val threats_path = spark.read.textFile("src/main/resources/cyber_threats").collect()
val dataSet = ???
dataSet.where(col("_source.raw_text").isin(thread_path: _*))
注意,如果thread_paths的大小很大,这将对性能产生影响,因为collect
和使用isin
的过滤器.
Note if the size of thread_paths is big, this will have performance impact both because of collect
and because of filter using isin
.
我建议您使用 join
过滤器 dataSet
和 threats_path
.它会像这样,
I'll suggest you to use filter dataSet
with threats_path
using join
. It will go something like,
val dataSet = spark.read.option("header","true").option("inferschema","true").json(path).na.drop
val threats_path = spark.read.textFile("src/main/resources/cyber_threats")
val newData = threats_path.join(dataSet, col("_source.raw_text") === col("<col in threats_path >"), "leftouter").show()
希望能帮到你
这篇关于基于Apache Spark中数组中的单词过滤DataFrame的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!