获取每组的第 20 到第 80 个百分位数 - Pyspark [英] Get 20th to 80th Percentile of each group - Pyspark
问题描述
我在 pyspark 数据框中有三列(下面给出了示例数据)
I have three columns in a pyspark data frame ( sample data given below )
订单类型 | 客户 ID | 金额 |
---|---|---|
A | c1 | 100.2 |
A | c2 | 1003.32 |
B | c1 | 222 |
C | c3 | 21.3 |
A | c4 | 1.2 |
我想从每个 orderType 中删除异常值.为了做到这一点,我从每个 orderType 的数据中删除了前 Nth Percentile.
I wanted to get the remove the outliers from each orderType. In order to do that I am removing the top Nth Percentile from the data for each orderType.
例如对于 N = 10,对于每个组,我将根据数量和 partitionBy orderType 获取第 10 到 90 个百分位数据.
For example for N = 10, for each group, I will fetch 10th to 90th Percentile data based on the amount and partitionBy orderType.
需要帮助为大型数据集(大约 6700 万行)实现该功能.
Need help to implement that for a large dataset ( around 67 million row count ) .
如果在这种情况下适用,也有人可以帮助在分区上使用近似分位数.
Also can someone help the possible usage of approxquantile on a partion if that is applicale in this case.
推荐答案
可以使用approx_percentile
,然后过滤:
import pyspark.sql.functions as F
df2 = df.withColumn(
'percentile',
F.expr("approx_percentile(amount, array(0.2, 0.8), 100) over (partition by orderType)")
).filter(
'amount between percentile[0] and percentile[1]'
)
在此处记录了该函数的用法.
Usage of the function is documented here.
这篇关于获取每组的第 20 到第 80 个百分位数 - Pyspark的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!