如何在可能为空的列上使用 PySpark CountVectorizer [英] How to use the PySpark CountVectorizer on columns that maybe null
问题描述
我的 Spark DataFrame 中有一个列:
I have a column in my Spark DataFrame:
|-- topics_A: array (nullable = true)
| |-- element: string (containsNull = true)
我正在使用 CountVectorizer:
I'm using CountVectorizer on it:
topic_vectorizer_A = CountVectorizer(inputCol="topics_A", outputCol="topics_vec_A")
我收到 NullPointerExceptions,因为有时 topic_A 列包含 null.
I get NullPointerExceptions, because sometimes the topic_A column contains null.
有没有办法解决这个问题?用零长度数组填充它可以正常工作(尽管它会大大增加数据大小) - 但我无法弄清楚如何在 PySpark 中的 Array 列上执行 fillNa.
Is there a way around this? Filling it with a zero-length array would work ok (although it will blow out the data size quite a lot) - but I can't work out how to do a fillNa on an Array column in PySpark.
推荐答案
就我个人而言,我会删除带有 NULL
值的列,因为那里没有有用的信息,但您可以用空数组替换空值.首先是一些进口:
Personally I would drop columns with NULL
values because there is no useful information there but you can replace nulls with empty arrays. First some imports:
from pyspark.sql.functions import when, col, coalesce, array
您可以将特定类型的空数组定义为:
You can define an empty array of specific type as:
fill = array().cast("array<string>")
并将其与 when
子句结合起来:
and combine it with when
clause:
topics_a = when(col("topics_A").isNull(), fill).otherwise(col("topics_A"))
或 coalesce
:
topics_a = coalesce(col("topics_A"), fill)
并将其用作:
df.withColumn("topics_A", topics_a)
使用示例数据:
df = sc.parallelize([(1, ["a", "b"]), (2, None)]).toDF(["id", "topics_A"])
df_ = df.withColumn("topics_A", topics_a)
topic_vectorizer_A.fit(df_).transform(df_)
结果将是:
+---+--------+-------------------+
| id|topics_A| topics_vec_A|
+---+--------+-------------------+
| 1| [a, b]|(2,[0,1],[1.0,1.0])|
| 2| []| (2,[],[])|
+---+--------+-------------------+
这篇关于如何在可能为空的列上使用 PySpark CountVectorizer的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!