pyspark 行号数据框 [英] pyspark row number dataframe

问题描述

我有一个数据框，包含时间、a、b、c、d、val 列.我想创建一个带有附加列的数据框，其中包含每个组中行的行号，其中 a、b、c、d 是组键.

I have a dataframe, with columns time,a,b,c,d,val. I would like to create a dataframe, with additional column, that will contain the row number of the row, within each group, where a,b,c,d is a group key.

我尝试使用 spark sql，通过定义一个窗口函数，特别是在 sql 中它看起来像这样:

I tried with spark sql, by defining a window function, in particular, in sql it will look like this:

select time, a,b,c,d,val, row_number() over(partition by a,b,c,d order by     time) as rn from table
group by a,b,c,d,val

我想在数据帧本身上执行此操作，而不使用 sparksql.

I would like to do this on the dataframe itslef, without using sparksql.

谢谢

推荐答案

python api 不太了解，但我会尝试一下.您可以尝试以下操作:

I don't know the python api too much, but I will give it a try. You can try something like:

from pyspark.sql import functions as F

df.withColumn("row_number", F.row_number().over(Window.partitionBy("a","b","c","d").orderBy("time"))).show()

这篇关于pyspark 行号数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！