Spark DataFrame组按降序排序(pyspark) [英] Spark DataFrame groupBy and sort in the descending order (pyspark)

问题描述

我正在使用pyspark(Python 2.7.9/Spark 1.3.1)，并且有一个数据框GroupObject，我需要对其进行过滤和过滤；降序排列.试图通过这段代码来实现它.

I'm using pyspark(Python 2.7.9/Spark 1.3.1) and have a dataframe GroupObject which I need to filter & sort in the descending order. Trying to achieve it via this piece of code.

group_by_dataframe.count().filter("`count` >= 10").sort('count', ascending=False)

但是会引发以下错误.

sort() got an unexpected keyword argument 'ascending'

推荐答案

在PySpark 1.3中，sort方法不采用升序参数.您可以改用desc方法:

In PySpark 1.3 sort method doesn't take ascending parameter. You can use desc method instead:

from pyspark.sql.functions import col

(group_by_dataframe
    .count()
    .filter("`count` >= 10")
    .sort(col("count").desc()))

或desc函数:

from pyspark.sql.functions import desc

(group_by_dataframe
    .count()
    .filter("`count` >= 10")
    .sort(desc("count"))

两种方法均可与Spark> = 1.3(包括Spark 2.x)一起使用.

Both methods can be used with with Spark >= 1.3 (including Spark 2.x).

这篇关于Spark DataFrame组按降序排序(pyspark)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！