如何在限制行数的同时拆分 Pyspark 数据帧? [英] How to split a Pyspark dataframe while limiting the number of rows?

问题描述

我正在将数据从数据帧发送到限制为 50,000 行的 API.假设我的数据帧有 70,000 行，我如何将其拆分为单独的数据帧，每个数据帧的最大行数为 50,000?这些不必是偶数，数据顺序无关紧要.

I am sending data from a dataframe to an API that has a limit of 50,000 rows. Say my dataframe has 70,000 rows, how can I split it into separate dataframes, each with a max row count of 50,000? These do not have to be even and the data order does not matter.

推荐答案

解决方法可以是使用 .limit() 函数.您可以执行以下操作:假设您的主 df 有 70k 行是 original_df.所以你可以像

Workaround for this can be to use .limit() function. You can do something like: let's say your main df with 70k rows is original_df. So you can do like

limited_df = df.limit(50000)

第一次获得 50k 行，对于接下来的行，您可以这样做

for the very first time to get the 50k rows and for the next rows you can do

original_df.subtract(limited_df)

您将获得剩余的行.如果需要，您甚至可以对减去的 df 执行 .limit() .

and you will get the remaining rows. You can even do .limit() for the subtracted df too if needed.

更新:您可以对数据框中存在的任意数量的行执行此操作.假设您的数据帧有 30000 行，如果您执行了 df.limit(50000)，它不会抛出任何错误，只会返回数据帧中存在的 30k 行.

UPDATED: You can do it for any number of rows present in dataframe. Let's say at one point if your dataframe has 30000 rows and if you did df.limit(50000), it's not gonna throw any error and will just return the 30k rows present in dataframe.

这篇关于如何在限制行数的同时拆分 Pyspark 数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！