将时间序列pySpark数据帧拆分为测试和在不使用随机分割的情况下进行训练 [英] Split Time Series pySpark data frame into test & train without using random split

问题描述

我有一个火花时间序列数据框.我想将其拆分为80-20(训练测试).由于这是一个时间序列数据帧，因此我不想进行随机拆分.为了将第一个数据帧传递到训练中并传递第二个数据帧进行测试，我该如何做?

I have a spark Time Series data frame. I would like to split it into 80-20 (train-test). As this is a time series data frame, I don't want to do a random split. How do I do this in order to pass the first data frame into train and the second to test?

推荐答案

您可以使用 pyspark.sql.functions.percent_rank() 以获得按时间戳/日期列排序的DataFrame的百分位排名.然后选择所有以rank <= 0.8作为训练集，其余作为测试集的列.

You can use pyspark.sql.functions.percent_rank() to get the percentile ranking of your DataFrame ordered by the timestamp/date column. Then pick all the columns with a rank <= 0.8 as your training set and the rest as your test set.

例如，如果您具有以下DataFrame:

For example, if you had the following DataFrame:

df.show(truncate=False)
#+---------------------+---+
#|date                 |x  |
#+---------------------+---+
#|2018-01-01 00:00:00.0|0  |
#|2018-01-02 00:00:00.0|1  |
#|2018-01-03 00:00:00.0|2  |
#|2018-01-04 00:00:00.0|3  |
#|2018-01-05 00:00:00.0|4  |
#+---------------------+---+

您需要训练集中的前4行和训练集中的最后一行.首先添加一列rank:

You'd want the first 4 rows in your training set and the last one in your training set. First add a column rank:

from pyspark.sql.functions import percent_rank
from pyspark.sql import Window

df = df.withColumn("rank", percent_rank().over(Window.partitionBy().orderBy("date")))

现在使用rank将数据拆分为train和test:

Now use rank to split your data into train and test:

train_df = df.where("rank <= .8").drop("rank")
train_df.show()
#+---------------------+---+
#|date                 |x  |
#+---------------------+---+
#|2018-01-01 00:00:00.0|0  |
#|2018-01-02 00:00:00.0|1  |
#|2018-01-03 00:00:00.0|2  |
#|2018-01-04 00:00:00.0|3  |
#+---------------------+---+

test_df = df.where("rank > .8").drop("rank")
test_df.show()
#+---------------------+---+
#|date                 |x  |
#+---------------------+---+
#|2018-01-05 00:00:00.0|4  |
#+---------------------+---+

这篇关于将时间序列pySpark数据帧拆分为测试和在不使用随机分割的情况下进行训练的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！