每组获得20％到80％的百分比-Pyspark [英] Get 20th to 80th Percentile of each group - Pyspark

问题描述

我在pyspark数据框中有三列(下面提供了示例数据)

I have three columns in a pyspark data frame ( sample data given below )

<身体>

orderType	customerId	金额
A	c1	100.2
A	c2	1003.32
B	c1	222
C	c3	21.3
A	c4	1.2

我想从每个orderType中删除异常值.为此，我从每个orderType的数据中删除了前N个百分位数.

I wanted to get the remove the outliers from each orderType. In order to do that I am removing the top Nth Percentile from the data for each orderType.

例如，对于N = 10，对于每个组，我将根据数量和partitionBy orderType来获取第10至第90个百分位数据.

For example for N = 10, for each group, I will fetch 10th to 90th Percentile data based on the amount and partitionBy orderType.

需要帮助为大型数据集(约6700万行计数)实现这一点.

Need help to implement that for a large dataset ( around 67 million row count ) .

如果在这种情况下适用，那么有人也可以帮助对一部分使用可能的分位数.

Also can someone help the possible usage of approxquantile on a partion if that is applicale in this case.

推荐答案

您可以使用 approx_percentile ，然后进行过滤:

You can use approx_percentile, then filter:

import pyspark.sql.functions as F

df2 = df.withColumn(
    'percentile',
    F.expr("approx_percentile(amount, array(0.2, 0.8), 100) over (partition by orderType)")
).filter(
    'amount between percentile[0] and percentile[1]'
)

在此处中记录了该功能的使用

Usage of the function is documented here.

这篇关于每组获得20％到80％的百分比-Pyspark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！