计算 PySpark DataFrame 列的模式? [英] Calculate the mode of a PySpark DataFrame column?

问题描述

最终我想要的是列的模式，对于 DataFrame 中的所有列.对于其他汇总统计，我看到了几个选项:使用 DataFrame 聚合，或将 DataFrame 的列映射到向量的 RDD(我也遇到了麻烦)并使用来自 MLlib 的 colStats.但我不认为模式是一种选择.

Ultimately what I want is the mode of a column, for all the columns in the DataFrame. For other summary statistics, I see a couple of options: use DataFrame aggregation, or map the columns of the DataFrame to an RDD of vectors (something I'm also having trouble doing) and use colStats from MLlib. But I don't see mode as an option there.

推荐答案

模式问题与中位数问题几乎相同.虽然它很容易计算，但计算相当昂贵.可以使用排序后跟本地和全局聚合或使用 just-another-wordcount 和过滤器来完成:

A problem with mode is pretty much the same as with median. While it is easy to compute, computation is rather expensive. It can be done either using sort followed by local and global aggregations or using just-another-wordcount and filter:

import numpy as np
np.random.seed(1)

df = sc.parallelize([
    (int(x), ) for x in np.random.randint(50, size=10000)
]).toDF(["x"])

cnts = df.groupBy("x").count()
mode = cnts.join(
    cnts.agg(max("count").alias("max_")), col("count") == col("max_")
).limit(1).select("x")
mode.first()[0]
## 0

无论哪种方式，每列都可能需要完全洗牌.

Either way it may require a full shuffle for each column.

这篇关于计算 PySpark DataFrame 列的模式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！