如何使用 DataFrames 在 PySpark 中使用窗口函数? [英] How to use window functions in PySpark using DataFrames?

问题描述

试图弄清楚如何在 PySpark 中使用窗口函数.这是我想要做的一个例子，简单地计算用户有一个事件"的次数(在这种情况下，dt"是一个模拟的时间戳).

Trying to figure out how to use window functions in PySpark. Here's an example of what I'd like to be able to do, simply count the number of times a user has an "event" (in this case "dt" is a simulated timestamp).

from pyspark.sql.window import Window
from pyspark.sql.functions import count

df = sqlContext.createDataFrame([{"id": 123, "dt": 0}, {"id": 123, "dt": 1}, {"id": 234, "dt":0}, {"id": 456, "dt":0}, {"id": 456, "dt":1}, {"id":456, "dt":2}])
df.select(["id","dt"], count("dt").over(Window.partitionBy("id").orderBy("dt")).alias("count")).show()

这会产生错误.使用窗口函数的正确方法是什么?我读到 1.4.1(我们需要使用的版本，因为它是 AWS 上的标准版本)应该能够使用 DataFrame API 来完成.

This produces an error. What is the correct way to use window functions? I read that 1.4.1 (the version we need to use since it's what is standard on AWS) should be able to do them with the DataFrame API.

FWIW，关于这个主题的文档非常少.而且我很难让任何示例实际运行.

FWIW, the documentation is pretty sparse on this subject. And I had trouble getting any examples actually running.

推荐答案

它抛出异常，因为您传递了一个列列表.DataFrame.select 的签名如下

It throws an exception because you pass a list of columns. Signature of DataFrame.select looks as follows

df.select(self, *cols)

并且使用窗口函数的表达式是一个与任何其他列一样的列，因此您在这里需要的是这样的:

and an expression using a window function is a column like any other so what you need here is something like this:

w = Window.partitionBy("id").orderBy("dt") # Just for clarity
df.select("id","dt", count("dt").over(w).alias("count")).show()

## +---+---+-----+
## | id| dt|count|
## +---+---+-----+
## |234|  0|    1|
## |456|  0|    1|
## |456|  1|    2|
## |456|  2|    3|
## |123|  0|    1|
## |123|  1|    2|
## +---+---+-----+

一般来说，Spark SQL 窗口函数的行为方式与任何现代 RDBMS 中的行为方式完全相同.

Generally speaking Spark SQL window functions behave exactly the same way as in any modern RDBMS.

这篇关于如何使用 DataFrames 在 PySpark 中使用窗口函数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！