火花数据帧分组不计算空值 [英] spark dataframe groupping does not count nulls

问题描述

我有一个spark数据框，它被分组为一个colum，并与一个计数聚合在一起：
df.groupBy（'a）.agg（count（a））。show

  + --------- + ---------------- + 
 | a | count（a）| 
 + --------- + ---------------- + 
 |空| 0 | 
 | -90 | 45684 | 
 + --------- + ---------------- + 
 
 
 df.select（'a ）.filter（'aisNull）.count 
  

返回

 警告：有一个功能警告;详细信息请使用-feature运行
 res9：Long = 26834 
  

最初不计算空值。

这种行为的原因是什么？我希望（如果 null 完全包含在分组结果中）正确查看计数。

解决方案

是的，应用于特定列的 count 不计算空值。如果您想包含空值，请使用：

  df.groupBy（'a）.agg（count（* ））。show 
  

I have a spark dataframe which is grouped ba a colum aggregated with a count: df.groupBy('a).agg(count("a")).show

+---------+----------------+
|a        |count(a)        |
+---------+----------------+
|     null|               0|
|      -90|           45684|
+---------+----------------+


df.select('a).filter('aisNull).count

returns

warning: there was one feature warning; re-run with -feature for details
res9: Long = 26834

which clearly shows that the null values were not counted initially.

What is the reason for this behaviour? I would have expected (if nullat all is contained in the grouping result) to properly see the counts.

解决方案

Yes, count applied to a specific column does not count the null-values. If you want to include the null-values, use:

df.groupBy('a).agg(count("*")).show

这篇关于火花数据帧分组不计算空值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！