Spark“用0替换空值"性能比较 [英] Spark "replacing null with 0" performance comparison

问题描述

Spark 1.6.1，Scala api.

Spark 1.6.1, Scala api.

对于数据框，我需要将某个列的所有空值替换为 0.我有两种方法可以做到这一点.1.

For a dataframe, I need to replace all null value of a certain column with 0. I have 2 ways to do this. 1.

myDF.withColumn("pipConfidence", when($"mycol".isNull, 0).otherwise($"mycol"))

2.

myDF.na.fill(0, Seq("mycol"))

它们本质上是相同的还是首选一种方式?

Are they essentially the same or one way is preferred?

谢谢！

推荐答案

它们不一样但性能应该相似.na.fill 使用 coalesce 但它取代了 NaN 和 NULLs，而不仅仅是 NULLS.

They are not the same but performance should be similar. na.fill uses coalesce but it replaces NaN and NULLs, not only NULLS.

val y = when($"x" === 0, $"x".cast("double")).when($"x" === 1, lit(null)).otherwise(lit("NaN").cast("double"))
val df = spark.range(0, 3).toDF("x").withColumn("y", y)

df.withColumn("y", when($"y".isNull, 0.0).otherwise($"y")).show()
df.na.fill(0.0, Seq("y")).show()

这篇关于Spark“用0替换空值"性能比较的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！