agg中count函数的pyspark-strange行为 [英] pyspark-strange behavior of count function inside agg

问题描述

我正在使用spark 2.4.0使用count函数进行汇总时，我观察到一种奇怪的行为.

I am using spark 2.4.0 I am observing a strange behavior while using count function to aggregate.

from pyspark.sql import functions as F
tst=sqlContext.createDataFrame([(1,2),(1,5),(2,None),(2,3),(3,None),(3,None)],schema=['col1','col2'])
tst.show()
+----+----+
|col1|col2|
+----+----+
|   1|   2|
|   1|   5|
|   2|null|
|   2|   3|
|   3|null|
|   3|null|
+----+----+

tst.groupby('col1').agg(F.count('col2')).show()
+----+-----------+
|col1|count(col2)|
+----+-----------+
|   1|          2|
|   3|          0|
|   2|          1|
+----+-----------+

在这里您可以看到不计算空值.我搜索了docus，但是没有提到函数计数不计算空值的地方.对我来说更令人惊讶的是

Here you can see that the null values are not counted. I searched for the docus, but no where it is mentioned that the function count does not count null values. More surprising for me is this

tst.groupby('col1').agg(F.count(F.col('col2').isNull())).show()
+----+---------------------+
|col1|count((col2 IS NULL))|
+----+---------------------+
|   1|                    2|
|   3|                    2|
|   2|                    2|
+----+---------------------+

在这里，我完全感到困惑.当我使用isNull()时，它不应该只计数空值吗?为什么要计算所有值?

Here I am totally confused. When I use isNull(), shouldn't it count only null values? Why is it counting all the values?

有什么我想念的吗?

推荐答案

在两种情况下，您看到的结果都是预期的结果.

In both cases the results that you see are the expected ones.

关于第一个示例:检查Scala

Concerning the first example: Checking the Scala source of count there is a subtle difference between count(*) and count('col2'):

FUNC (*)-返回已检索的行总数，包括包含null的行.
FUNC (expr [，expr ...])-返回所提供的表达式都不为空的行数.

FUNC(*) - Returns the total number of retrieved rows, including rows containing null.
FUNC(expr[, expr...]) - Returns the number of rows for which the supplied expression(s) are all non-null.

这说明了为什么不计算 null 项的原因.

This explains why the null entries are not counted.

如果您将代码更改为

tst.groupby('col1').agg(F.count('*')).show()

你得到

+----+--------+
|col1|count(1)|
+----+--------+
|   1|       2|
|   3|       2|
|   2|       2|
+----+--------+

---

关于第二部分:表达式 F.col('col2').isNull()返回布尔值.无论此布尔值的实际值是多少，都将对行进行计数，因此您会看到 2 .

---

About the second part: the expression F.col('col2').isNull() returns a boolean value. No matter what the actual value of this boolean is, the row is counted and therefore you see a 2.

这篇关于agg中count函数的pyspark-strange行为的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！