Spark数据框选择在该行的任何列中至少包含一个null或空白的行 [英] Spark dataframe select rows with at least one null or blank in any column of that row

问题描述

从一个数据帧中我想创建一个新的数据帧，其中任何列中的至少一个值在spark 1.5/scala中为null或为空白.

我正在尝试编写一个泛化函数来创建此新数据框.我在其中传递数据框和列列表并创建记录.

谢谢

解决方案

样本数据:

val df = Seq((null, Some(2)), (Some("a"), Some(4)), (Some(""), Some(5)), (Some("b"), null)).toDF("A", "B")

df.show
+----+----+
|   A|   B|
+----+----+
|null|   2|
|   a|   4|
|    |   5|
|   b|null|
+----+----+  

您可以将条件构造为，假设空白表示此处为空字符串:

import org.apache.spark.sql.functions.col
val cond = df.columns.map(x => col(x).isNull || col(x) === "").reduce(_ || _)

df.filter(cond).show
+----+----+
|   A|   B|
+----+----+
|null|   2|
|    |   5|
|   b|null|
+----+----+

from one dataframe i want to create a new dataframe where at least one value in any of the columns is null or blank in spark 1.5 / scala.

i am trying to write a generalize function to create this new dataframe. where i pass the dataframe and the list of columns and creates the record.

Thanks

解决方案

Sample Data:

val df = Seq((null, Some(2)), (Some("a"), Some(4)), (Some(""), Some(5)), (Some("b"), null)).toDF("A", "B")

df.show
+----+----+
|   A|   B|
+----+----+
|null|   2|
|   a|   4|
|    |   5|
|   b|null|
+----+----+  

You can construct the condition as, assume blank means empty string here:

import org.apache.spark.sql.functions.col
val cond = df.columns.map(x => col(x).isNull || col(x) === "").reduce(_ || _)

df.filter(cond).show
+----+----+
|   A|   B|
+----+----+
|null|   2|
|    |   5|
|   b|null|
+----+----+

这篇关于Spark数据框选择在该行的任何列中至少包含一个null或空白的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！