使用 Spark (1.6) 从 Scala 数据框中的数组列中删除空值 [英] Remove Null from Array Columns in Dataframe in Scala with Spark (1.6)
问题描述
我有一个数据框,其中包含一个键列和一个包含结构数组的列.架构如下所示.
I have a dataframe with a key column and a column which has an array of struct. The Schema looks like below.
root
|-- id: string (nullable = true)
|-- desc: array (nullable = false)
| |-- element: struct (containsNull = true)
| | |-- name: string (nullable = true)
| | |-- age: long (nullable = false)
数组desc"可以有任意数量的空值.我想使用 spark 1.6 创建一个没有空值的数组的最终数据帧:
The array "desc" can have any number of null values. I would like to create a final dataframe with the array having none of the null values using spark 1.6:
一个例子是:
Key . Value
1010 . [[George,21],null,[MARIE,13],null]
1023 . [null,[Watson,11],[John,35],null,[Kyle,33]]
我希望最终的数据帧为:
I want the final dataframe as:
Key . Value
1010 . [[George,21],[MARIE,13]]
1023 . [[Watson,11],[John,35],[Kyle,33]]
我尝试使用 UDF 和 case 类来执行此操作,但得到了
I tried doing this with UDF and case class but got
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to....
非常感谢任何帮助,如果需要,我更愿意在不转换为 RDD 的情况下进行.我也是 spark 和 scala 的新手,所以提前致谢!!!
Any help is greatly appreciated and I would prefer doing it without converting to RDDs if needed. Also I am new to spark and scala so thanks in advance!!!
推荐答案
鉴于原始数据帧具有以下架构
Given that the original dataframe has following schema
root
|-- id: string (nullable = true)
|-- desc: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- name: string (nullable = true)
| | |-- age: long (nullable = false)
定义一个udf
函数从数组中删除空值应该对你有用
import org.apache.spark.sql.functions._
def removeNull = udf((array: Seq[Row])=> array.filterNot(_ == null).map(x => element(x.getAs[String]("name"), x.getAs[Long]("age"))))
df.withColumn("desc", removeNull(col("desc")))
其中 element
是一个 case 类
case class element(name: String, age: Long)
你应该得到
+----+-----------------------------------+
|id |desc |
+----+-----------------------------------+
|1010|[[George,21], [MARIE,13]] |
|1010|[[Watson,11], [John,35], [Kyle,33]]|
+----+-----------------------------------+
这篇关于使用 Spark (1.6) 从 Scala 数据框中的数组列中删除空值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!