Pyspark:如何计算单个群组的平均和计数? [英] Pyspark:How to calculate avg and count in a single groupBy?

问题描述

我想通过Pyspark中的语句来计算平均值并在单个组中进行计数.我该怎么办?

I would like to calculate avg and count in a single group by statement in Pyspark. How can I do that?

df = spark.createDataFrame([(1, 'John', 1.79, 28,'M', 'Doctor'),
                        (2, 'Steve', 1.78, 45,'M', None),
                        (3, 'Emma', 1.75, None, None, None),
                        (4, 'Ashley',1.6, 33,'F', 'Analyst'),
                        (5, 'Olivia', 1.8, 54,'F', 'Teacher'),
                        (6, 'Hannah', 1.82, None, 'F', None),
                        (7, 'William', 1.7, 42,'M', 'Engineer'),
                        (None,None,None,None,None,None),
                        (8,'Ethan',1.55,38,'M','Doctor'),
                        (9,'Hannah',1.65,None,'F','Doctor')]
                       , ['Id', 'Name', 'Height', 'Age', 'Gender', 'Profession'])

#这仅显示avg，但我需要在其旁边进行计数.我该怎么办?

#This only shows avg but also I need count right next to it. How can I do that?

df.groupBy("Profession").agg({"Age":"avg"}).show()
df.show()

谢谢.

推荐答案

对于同一列:

from pyspark.sql import functions as F
df.groupBy("Profession").agg(F.mean('Age'), F.count('Age')).show()

如果您可以使用其他列:

If you're able to use different columns:

df.groupBy("Profession").agg({'Age':'avg', 'Gender':'count'}).show()

这篇关于Pyspark:如何计算单个群组的平均和计数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！