将多个groupBy函数合并为1 [英] Combining multiple groupBy functions into 1
本文介绍了将多个groupBy函数合并为1的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
使用此代码查找模式:
import numpy as np
np.random.seed(1)
df2 = sc.parallelize([
(int(x), ) for x in np.random.randint(50, size=10000)
]).toDF(["x"])
cnts = df2.groupBy("x").count()
mode = cnts.join(
cnts.agg(max("count").alias("max_")), col("count") == col("max_")
).limit(1).select("x")
mode.first()[0]
返回错误:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-53-2a9274e248ac> in <module>()
8 cnts = df.groupBy("x").count()
9 mode = cnts.join(
---> 10 cnts.agg(max("count").alias("max_")), col("count") == col("max_")
11 ).limit(1).select("x")
12 mode.first()[0]
AttributeError: 'str' object has no attribute 'alias'
我正在尝试使用此自定义方法:
Instead of this solution I'm attempting this custom one:
df.show()
cnts = df.groupBy("c1").count()
print cnts.rdd.map(tuple).sortBy(lambda a: a[1], ascending=False).first()
cnts = df.groupBy("c2").count()
print cnts.rdd.map(tuple).sortBy(lambda a: a[1] , ascending=False).first()
返回:
c1
&的模态c2
分别是2.0和3.0
So modal of c1
& c2
are 2.0 and 3.0 respectively
这可以应用于数据框中的所有列c1,c2,c3,c4,c5
,而不是像我所做的那样显式选择每个列吗?
Can this be applied to all columns c1,c2,c3,c4,c5
in dataframe instead of explicitly selecting each column as I have done ?
推荐答案
似乎您正在使用内置的max
,而不是SQL函数.
It looks like you're using built-in max
, not a SQL function.
import pyspark.sql.functions as F
cnts.agg(F.max("count").alias("max_"))
要在同一类型的多列上查找模式,可以将其整形为long(如 Apache中的熊猫融化函数所定义的melt
Spark ):
To find mode over multiple columns of the same type you can reshape to long (melt
as defined in Pandas Melt function in Apache Spark):
(melt(df, [], df.columns)
# Count by column and value
.groupBy("variable", "value")
.count()
# Find mode per column
.groupBy("variable")
.agg(F.max(F.struct("count", "value")).alias("mode"))
.select("variable", "mode.value"))
+--------+-----+
|variable|value|
+--------+-----+
| c5| 6.0|
| c1| 2.0|
| c4| 5.0|
| c3| 4.0|
| c2| 3.0|
+--------+-----+
这篇关于将多个groupBy函数合并为1的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文