如何使用应替换的空值(用 0)最大化每列? [英] How to max per column with nulls that should be replaced (with 0)?
本文介绍了如何使用应替换的空值(用 0)最大化每列?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
如何获得下面数据框中的 MAX?
How to get the MAX in the below dataframe?
val df_n = df.select($"ID").filter(($"READ") === "" && ($"ACT"!==""))
我必须找出ID
的最大值,如果ID
是NULL
,我必须用0替换它.
I have to find out the MAX of ID
and in case if ID
is NULL
, I have to replace it with 0.
推荐答案
以下内容如何?
scala> val df = Seq("0", null, "5", null, null, "-8").toDF("id")
df: org.apache.spark.sql.DataFrame = [id: string]
scala> df.printSchema
root
|-- id: string (nullable = true)
scala> df.withColumn("idAsLong", $"id" cast "long").printSchema
root
|-- id: string (nullable = true)
|-- idAsLong: long (nullable = true)
scala> val testDF = df.withColumn("idAsLong", $"id" cast "long")
testDF: org.apache.spark.sql.DataFrame = [id: string, idAsLong: bigint]
scala> testDF.show
+----+--------+
| id|idAsLong|
+----+--------+
| 0| 0|
|null| null|
| 5| 5|
|null| null|
|null| null|
| -8| -8|
+----+--------+
解决方案
scala> testDF.agg(max("idAsLong")).show
+-------------+
|max(idAsLong)|
+-------------+
| 5|
+-------------+
使用 na 运算符
如果只有负值和 null
并且 null
是最大值怎么办?在 Dataset
上使用 na
运算符.
Using na Operator
What if you had only negative values and null
and so null
is the maximum value? Use na
operator on Dataset
.
val withNulls = Seq("-1", "-5", null, null, "-333", null)
.toDF("id")
.withColumn("asInt", $"id" cast "int") // <-- column of type int with nulls
scala> withNulls.na.fill(Map("asInt" -> 0)).agg(max("asInt")).show
+----------+
|max(asInt)|
+----------+
| 0|
+----------+
如果没有 na
和替换 null
,它根本就行不通.
Without na
and replacing null
it simply won't work.
scala> withNulls.agg(max("asInt")).show
+----------+
|max(asInt)|
+----------+
| -1|
+----------+
这篇关于如何使用应替换的空值(用 0)最大化每列?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文