有没有办法获取Spark数据帧的前1000行? [英] Is there a way to take the first 1000 rows of a Spark Dataframe?

问题描述

我正在使用randomSplit函数来获取少量数据帧以用于开发目的，而我最终只是获取了此函数返回的第一个df.

I am using the randomSplitfunction to get a small amount of a dataframe to use in dev purposes and I end up just taking the first df that is returned by this function.

val df_subset = data.randomSplit(Array(0.00000001, 0.01), seed = 12345)(0)

如果我使用df.take(1000)，那么我最终将得到一个行数组-而不是一个数据框，因此这对我不起作用.

If I use df.take(1000) then I end up with an array of rows- not a dataframe, so that won't work for me.

是否有更好，更简单的方法来获取df的前1000行并将其存储为另一个df?

Is there a better, simpler way to take say the first 1000 rows of the df and store it as another df?

推荐答案

您正在寻找的方法是

通过获取前n行来返回新的数据集.此函数与head的区别在于head返回一个数组，而limit返回一个新的数据集.

Returns a new Dataset by taking the first n rows. The difference between this function and head is that head returns an array while limit returns a new Dataset.

这篇关于有没有办法获取Spark数据帧的前1000行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！