如何选择最后一行以及如何通过索引访问 PySpark 数据框? [英] How to select last row and also how to access PySpark dataframe by index?

问题描述

来自像

name age city
abc   20  A
def   30  B

如何获取最后一行.(就像通过 df.limit(1) 我可以将数据帧的第一行放入新的数据帧中).

How to get the last row.(Like by df.limit(1) I can get first row of dataframe into new dataframe).

以及如何通过 index.like 行号访问数据帧行.12 或 200 .

And how can I access the dataframe rows by index.like row no. 12 or 200 .

在熊猫中我可以做到

df.tail(1) # for last row
df.ix[rowno or index] # by index
df.loc[] or by df.iloc[]

我只是好奇如何以这种方式或其他方式访问 pyspark 数据框.

I am just curious how to access pyspark dataframe in such ways or alternative ways.

谢谢

推荐答案

如何获取最后一行.

How to get the last row.

假设所有列都可以排序的冗长而丑陋的方式:

Long and ugly way which assumes that all columns are oderable:

from pyspark.sql.functions import (
    col, max as max_, struct, monotonically_increasing_id
)

last_row = (df
    .withColumn("_id", monotonically_increasing_id())
    .select(max(struct("_id", *df.columns))
    .alias("tmp")).select(col("tmp.*"))
    .drop("_id"))

如果不是所有列都可以排序，您可以尝试:

If not all columns can be order you can try:

with_id = df.withColumn("_id", monotonically_increasing_id())
i = with_id.select(max_("_id")).first()[0]

with_id.where(col("_id") == i).drop("_id")

注意.pyspark.sql.functions/`oassql.functions 中有 last 函数，但考虑 描述对应的表达式这里不是一个好的选择.

Note. There is last function in pyspark.sql.functions/ `o.a.s.sql.functions but considering description of the corresponding expressions it is not a good choice here.

如何通过 index.like 访问数据帧行

how can I access the dataframe rows by index.like

你不能.Spark DataFrame 可通过索引访问.您可以使用 zipWithIndex 添加索引并稍后过滤.请记住这个 O(N) 操作.

You cannot. Spark DataFrame and accessible by index. You can add indices using zipWithIndex and filter later. Just keep in mind this O(N) operation.

这篇关于如何选择最后一行以及如何通过索引访问 PySpark 数据框?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！