如何在PySpark中将字符串值替换为NULL? [英] How do I replace a string value with a NULL in PySpark?

问题描述

我想做这样的事情:

df.replace('empty-value', None, 'NAME')

基本上，我想用NULL代替一些值.但此功能不接受无".我该怎么办?

Basically, I want to replace some value with NULL. but it does not accept None in this function. How can I do this?

推荐答案

这将在您的name列中将empty-value替换为None:

This will replace empty-value with None in your name column:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType


df = sc.parallelize([(1, "empty-value"), (2, "something else")]).toDF(["key", "name"])
new_column_udf = udf(lambda name: None if name == "empty-value" else name, StringType())
new_df = df.withColumn("name", new_column_udf(df.name))
new_df.collect()

输出:

[Row(key=1, name=None), Row(key=2, name=u'something else')]

通过使用旧名称作为withColumn中的第一个参数，它实际上将旧的name列替换为UDF输出生成的新列.

By using the old name as the first parameter in withColumn, it actually replaces the old name column with the new one generated by the UDF output.

这篇关于如何在PySpark中将字符串值替换为NULL?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！