pyspark行号数据框 [英] pyspark row number dataframe

问题描述

我有一个数据框，其中包含时间time，a，b，c，d，val. 我想创建一个带有附加列的数据框，该数据框将包含每个组内行的行号，其中a，b，c，d是组键.

I have a dataframe, with columns time,a,b,c,d,val. I would like to create a dataframe, with additional column, that will contain the row number of the row, within each group, where a,b,c,d is a group key.

我尝试通过定义窗口函数来使用spark sql，特别是在sql中，它看起来像这样:

I tried with spark sql, by defining a window function, in particular, in sql it will look like this:

select time, a,b,c,d,val, row_number() over(partition by a,b,c,d order by     time) as rn from table
group by a,b,c,d,val

我想在数据框itslef上执行此操作，而无需使用sparksql.

I would like to do this on the dataframe itslef, without using sparksql.

谢谢

推荐答案

我不太了解python api，但我会尝试一下.您可以尝试以下操作:

I don't know the python api too much, but I will give it a try. You can try something like:

from pyspark.sql import functions as F

df.withColumn("row_number", F.row_number().over(Window.partitionBy("a","b","c","d").orderBy("time"))).show()

这篇关于pyspark行号数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！