通过检查字符串是否出现在列中来过滤PySpark DataFrame [英] Filter PySpark DataFrame by checking if string appears in column

问题描述

我是Spark的新手，正在玩过滤.我有一个通过读取json文件创建的pyspark.sql DataFrame.模式的一部分如下所示:

I'm new to Spark and playing around with filtering. I have a pyspark.sql DataFrame created by reading in a json file. A part of the schema is shown below:

root
 |-- authors: array (nullable = true)
 |    |-- element: string (containsNull = true)

我想过滤此DataFrame，选择所有与特定作者相关的行.因此，无论该作者是authors中列出的第一位作者还是第n位，如果出现其姓名，都应包括该行.因此，类似

I would like to filter this DataFrame, selecting all of the rows with entries pertaining to a particular author. So whether this author is the first author listed in authors or the nth, the row should be included if their name appears. So something along the lines of

df.filter(df['authors'].getItem(i)=='Some Author')

其中i遍历该行中的所有作者，但在各行中不是恒定的.

where i iterates through all authors in that row, which is not constant across rows.

我尝试实施为 PySpark DataFrames提供的解决方案:过滤数组列中某些值的地方，但是它给了我

I tried implementing the solution given to PySpark DataFrames: filter where some value is in array column, but it gives me

ValueError::某些类型不能由前100行确定， 请尝试重试

ValueError: Some of types cannot be determined by the first 100 rows, please try again with sampling

是否有实现此过滤器的简洁方法?

Is there a succinct way to implement this filter?

推荐答案

您可以使用pyspark.sql.functions.array_contains方法:

df.filter(array_contains(df['authors'], 'Some Author'))

---

from pyspark.sql.types import *
from pyspark.sql.functions import array_contains

lst = [(["author 1", "author 2"],), (["author 2"],) , (["author 1"],)]
schema = StructType([StructField("authors", ArrayType(StringType()), True)])
df = spark.createDataFrame(lst, schema)
df.show()
+--------------------+
|             authors|
+--------------------+
|[author 1, author 2]|
|          [author 2]|
|          [author 1]|
+--------------------+

df.printSchema()
root
 |-- authors: array (nullable = true)
 |    |-- element: string (containsNull = true)

df.filter(array_contains(df.authors, "author 1")).show()
+--------------------+
|             authors|
+--------------------+
|[author 1, author 2]|
|          [author 1]|
+--------------------+

这篇关于通过检查字符串是否出现在列中来过滤PySpark DataFrame的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！