通过检查字符串是否出现在列中来过滤 PySpark DataFrame [英] Filter PySpark DataFrame by checking if string appears in column

问题描述

我是 Spark 的新手，正在尝试过滤.我有一个通过读取 json 文件创建的 pyspark.sql DataFrame.部分架构如下所示:

I'm new to Spark and playing around with filtering. I have a pyspark.sql DataFrame created by reading in a json file. A part of the schema is shown below:

root
 |-- authors: array (nullable = true)
 |    |-- element: string (containsNull = true)

我想过滤此 DataFrame，选择包含与特定作者有关的条目的所有行.因此，无论此作者是 authors 中列出的第一作者还是第 n 位，如果出现他们的姓名，则应包括该行.所以类似

I would like to filter this DataFrame, selecting all of the rows with entries pertaining to a particular author. So whether this author is the first author listed in authors or the nth, the row should be included if their name appears. So something along the lines of

df.filter(df['authors'].getItem(i)=='Some Author')

其中 i 遍历该行中的所有作者，这在行之间不是恒定的.

where i iterates through all authors in that row, which is not constant across rows.

我尝试实施给 PySpark DataFrames 的解决方案: 过滤数组列中的某个值，但它给了我

I tried implementing the solution given to PySpark DataFrames: filter where some value is in array column, but it gives me

ValueError: 某些类型无法通过前 100 行确定，请再试一次采样

ValueError: Some of types cannot be determined by the first 100 rows, please try again with sampling

有没有简洁的方法来实现这个过滤器?

Is there a succinct way to implement this filter?

推荐答案

您可以使用 pyspark.sql.functions.array_contains 方法:

df.filter(array_contains(df['authors'], 'Some Author'))

<小时>

from pyspark.sql.types import *
from pyspark.sql.functions import array_contains

lst = [(["author 1", "author 2"],), (["author 2"],) , (["author 1"],)]
schema = StructType([StructField("authors", ArrayType(StringType()), True)])
df = spark.createDataFrame(lst, schema)
df.show()
+--------------------+
|             authors|
+--------------------+
|[author 1, author 2]|
|          [author 2]|
|          [author 1]|
+--------------------+

df.printSchema()
root
 |-- authors: array (nullable = true)
 |    |-- element: string (containsNull = true)

df.filter(array_contains(df.authors, "author 1")).show()
+--------------------+
|             authors|
+--------------------+
|[author 1, author 2]|
|          [author 1]|
+--------------------+

这篇关于通过检查字符串是否出现在列中来过滤 PySpark DataFrame的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！