如何选择最后一行以及如何按索引访问PySpark数据帧? [英] How to select last row and also how to access PySpark dataframe by index?

问题描述

从

name age city
abc   20  A
def   30  B

如何获取最后一行.(像df.limit(1)一样，我可以将数据帧的第一行放入新的数据帧中.)

How to get the last row.(Like by df.limit(1) I can get first row of dataframe into new dataframe).

又如何按索引访问数据框行，如行号. 12或200.

And how can I access the dataframe rows by index.like row no. 12 or 200 .

我可以在熊猫里做

df.tail(1) # for last row
df.ix[rowno or index] # by index
df.loc[] or by df.iloc[]

我很好奇如何以这种方式或其他方式访问pyspark数据框.

I am just curious how to access pyspark dataframe in such ways or alternative ways.

谢谢

推荐答案

如何获取最后一行.

How to get the last row.

漫长而丑陋的方式假设所有列都是可修改的:

Long and ugly way which assumes that all columns are oderable:

from pyspark.sql.functions import (
    col, max as max_, struct, monotonically_increasing_id
)

last_row = (df
    .withColumn("_id", monotonically_increasing_id())
    .select(max(struct("_id", *df.columns))
    .alias("tmp")).select(col("tmp.*"))
    .drop("_id"))

如果不是所有列都可以排序，则可以尝试:

If not all columns can be order you can try:

with_id = df.withColumn("_id", monotonically_increasing_id())
i = with_id.select(max_("_id")).first()[0]

with_id.where(col("_id") == i).drop("_id")

注意.在pyspark.sql.functions/`oassql.functions中有last函数，但是正在考虑

Note. There is last function in pyspark.sql.functions/ `o.a.s.sql.functions but considering description of the corresponding expressions it is not a good choice here.

如何通过index.like

how can I access the dataframe rows by index.like

您不能. Spark DataFrame，可通过索引访问. 您可以使用zipWithIndex 添加索引，并在以后进行过滤.请记住此 O(N)操作.

You cannot. Spark DataFrame and accessible by index. You can add indices using zipWithIndex and filter later. Just keep in mind this O(N) operation.

这篇关于如何选择最后一行以及如何按索引访问PySpark数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！