Pyspark 用 NULL 替换 NaN [英] Pyspark replace NaN with NULL
问题描述
我使用 Spark 执行加载到 Redshift 中的数据转换.Redshift 不支持 NaN 值,所以我需要用 NULL 替换所有出现的 NaN.
I use Spark to perform data transformations that I load into Redshift. Redshift does not support NaN values, so I need to replace all occurrences of NaN with NULL.
我尝试过这样的事情:
some_table = sql('SELECT * FROM some_table')
some_table = some_table.na.fill(None)
但我收到以下错误:
ValueError: value 应该是 float、int、long、string、bool 或 dict
ValueError: value should be a float, int, long, string, bool or dict
所以看起来 na.fill()
不支持 None.我特别需要替换为 NULL
,而不是其他一些值,例如 0
.
So it seems like na.fill()
doesn't support None. I specifically need to replace with NULL
, not some other value, like 0
.
推荐答案
我在谷歌上搜索了一下,终于找到了答案.
I finally found the answer after Googling around a bit.
df = spark.createDataFrame([(1, float('nan')), (None, 1.0)], ("a", "b"))
df.show()
+----+---+
| a| b|
+----+---+
| 1|NaN|
|null|1.0|
+----+---+
import pyspark.sql.functions as F
columns = df.columns
for column in columns:
df = df.withColumn(column,F.when(F.isnan(F.col(column)),None).otherwise(F.col(column)))
sqlContext.registerDataFrameAsTable(df, "df2")
sql('select * from df2').show()
+----+----+
| a| b|
+----+----+
| 1|null|
|null| 1.0|
+----+----+
它没有使用 na.fill()
,但它实现了相同的结果,所以我很高兴.
It doesn't use na.fill()
, but it accomplished the same result, so I'm happy.
这篇关于Pyspark 用 NULL 替换 NaN的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!