pyspark -- 对 Array(Integer()) 类型的列中的值求和的最佳方法 [英] pyspark -- best way to sum values in column of type Array(Integer())

问题描述

假设这是我的数据框...

Lets say this is my dataframe ...

name | scores
Dan  |  [10,5,2,12]
Ann  |  [ 12,3,5]
Jon  |  [ ] 

所需的输出类似于

name | scores         | Total
Dan  |  [10,5,2,12]   | 29
Ann  |   [ 12,3,5]    | 20
Jon  |  [ ]           | 0

我按照......制作了一个 UDF

I made a UDF along the lines of ....

sum_cols = udf(lambda arr: if arr == [] then 0 else __builtins__.sum(arr),IntegerType())

df.withColumn('Total', sum_cols(col('scores'))).show()

但是，我了解到 UDF 相对于纯 pySpark 函数来说相对较慢.

However, I have learned that UDFs are relatively slow to pure pySpark functions.

有没有办法在没有 UDF 的情况下在 pySpark 中执行上面的代码?

Any way to do code above in pySpark without a UDF ?

推荐答案

可以使用高阶 SQL 函数 AGGREGATE(从函数式编程中减少)，像这样:

You can use a higher-order SQL function AGGREGATE (reduce from functional programming), like this:

import pyspark.sql.functions as F
df = df.select(
  'name',
  F.expr('AGGREGATE(scores, 0, (acc, x) -> acc + x)').alias('Total')
)

第一个参数是数组列，第二个是初始值(应该与您求和的值具有相同的类型，因此如果您的输入不是整数，您可能需要使用0.0"或DOUBLE(0)"等)第三个参数是一个 lambda 函数，它将数组的每个元素添加到一个累加器变量中(一开始这将被设置为初始值 0).

First argument is the array column, second is initial value (should be of same type as the values you sum, so you may need to use "0.0" or "DOUBLE(0)" etc if your inputs are not integers) and third argument is a lambda function, which adds each element of the array to an accumulator variable (in the beginning this will be set to the initial value 0).

转换将在单个投影算子中运行，因此将非常有效.此外，您不需要提前知道数组的大小，数组的每一行可以有不同的长度.

The transformation will run in a single projection operator, thus will be very efficient. Also you do not need to know the size of the arrays in advance and the array can have different length on each row.

这篇关于pyspark -- 对 Array(Integer()) 类型的列中的值求和的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！