spark数据帧分组不计算空值 [英] spark dataframe groupping does not count nulls

问题描述

我有一个 spark DataFrame，它按与计数聚合的列分组:

I have a spark DataFrame which is grouped by a column aggregated with a count:

df.groupBy('a').agg(count("a")).show

+---------+----------------+
|a        |count(a)        |
+---------+----------------+
|     null|               0|
|      -90|           45684|
+---------+----------------+


df.select('a').filter('aisNull').count

返回

warning: there was one feature warning; re-run with -feature for details
res9: Long = 26834

这清楚地表明最初没有计算空值.

which clearly shows that the null values were not counted initially.

这种行为的原因是什么?我会期望(如果 null 根本包含在分组结果中)正确地看到计数.

What is the reason for this behaviour? I would have expected (if nullat all is contained in the grouping result) to properly see the counts.

推荐答案

是的，应用于特定列的 count 不计算空值.如果要包含空值，请使用:

Yes, count applied to a specific column does not count the null-values. If you want to include the null-values, use:

df.groupBy('a).agg(count("*")).show

这篇关于spark数据帧分组不计算空值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！