在多列上应用窗口函数 [英] Apply window function over multiple columns
本文介绍了在多列上应用窗口函数的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想执行窗口函数(具体来说是移动平均线),但要在数据帧的所有列上执行.
I would like to perform window function (concretely moving average), but over all columns of a dataframe.
我可以这样做
from pyspark.sql import SparkSession, functions as func
df = ...
df.select([func.avg(df[col]).over(windowSpec).alias(col) for col in df.columns])
但恐怕这不是很有效.有没有更好的方法?
but I'm afraid this isn't very efficient. Is there a better way to do it?
推荐答案
另一种可能更好的替代方法是创建一个新的 df,您可以在其中按 Window 函数中的列分组并对其余列应用平均值,然后进行左连接.对于 df 溢出到磁盘(或无法保存在内存中)的大型数据帧,这肯定会更优化.
An alternative which may be better is to create a new df where you Group By the columns in Window function and apply average on the remaining columns then do a left join. For large data frames where the df is being spilled over to disk (or cannot be persisted in memory), this will definitely be more optimal.
这篇关于在多列上应用窗口函数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文