分组 PySpark DataFrame 后如何应用描述函数? [英] How to apply the describe function after grouping a PySpark DataFrame?

问题描述

我想找到将 describe 函数应用到分组数据帧的最简洁的方法(这个问题也可以扩展到将任何 DF 函数应用到分组 DF)

I want to find the cleanest way to apply the describe function to a grouped DataFrame (this question can also grow to apply any DF function to a grouped DF)

我在没有运气的情况下测试了分组聚合 Pandas UDF.总有一种方法可以通过在 agg 函数中传递每个统计信息来实现，但这不是正确的方法.

I tested grouped aggregate pandas UDF with no luck. There's always a way of doing it by passing each statistics inside the agg function but that's not the proper way.

如果我们有一个示例数据框:

If we have a sample dataframe:

df = spark.createDataFrame(
    [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
    ("id", "v"))

我们的想法是做一些类似于 Pandas 的事情:

The idea would be to do something similar to Pandas:

df.groupby("id").describe()

结果在哪里:

                                                   v
    count mean     std    min   25%    50%  75%   max
id                              
1   2.0   1.5   0.707107  1.0   1.25   1.5  1.75  2.0
2   3.0   6.0   3.605551  3.0   4.00   5.0  7.50  10.0

谢谢.

推荐答案

试试这个:

df.groupby("id").agg(F.count('v').alias('count'), F.mean('v').alias('mean'), F.stddev('v').alias('std'), F.min('v').alias('min'), F.expr('percentile(v, array(0.25))')[0].alias('%25'),  F.expr('percentile(v, array(0.5))')[0].alias('%50'), F.expr('percentile(v, array(0.75))')[0].alias('%75'), F.max('v').alias('max')).show()

输出:

+---+-----+----+------------------+---+----+---+----+----+
| id|count|mean|               std|min| %25|%50| %75| max|
+---+-----+----+------------------+---+----+---+----+----+
|  1|    2| 1.5|0.7071067811865476|1.0|1.25|1.5|1.75| 2.0|
|  2|    3| 6.0| 3.605551275463989|3.0| 4.0|5.0| 7.5|10.0|
+---+-----+----+------------------+---+----+---+----+----+

这篇关于分组 PySpark DataFrame 后如何应用描述函数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！