将时间序列 pySpark 数据帧拆分为 test &不使用随机拆分训练 [英] Split Time Series pySpark data frame into test & train without using random split
问题描述
我有一个 spark 时间序列数据框.我想把它分成 80-20 个(训练测试).由于这是一个时间序列数据框,我不想进行随机拆分.我该怎么做才能将第一个数据帧传递到训练中,然后将第二个数据帧传递到测试中?
I have a spark Time Series data frame. I would like to split it into 80-20 (train-test). As this is a time series data frame, I don't want to do a random split. How do I do this in order to pass the first data frame into train and the second to test?
推荐答案
您可以使用pyspark.sql.functions.percent_rank()
以获取按时间戳/日期列排序的 DataFrame 的百分位排名.然后选择所有 rank <= 0.8
的列作为你的训练集,其余的作为你的测试集.
You can use pyspark.sql.functions.percent_rank()
to get the percentile ranking of your DataFrame ordered by the timestamp/date column. Then pick all the columns with a rank <= 0.8
as your training set and the rest as your test set.
例如,如果您有以下 DataFrame:
For example, if you had the following DataFrame:
df.show(truncate=False)
#+---------------------+---+
#|date |x |
#+---------------------+---+
#|2018-01-01 00:00:00.0|0 |
#|2018-01-02 00:00:00.0|1 |
#|2018-01-03 00:00:00.0|2 |
#|2018-01-04 00:00:00.0|3 |
#|2018-01-05 00:00:00.0|4 |
#+---------------------+---+
您需要训练集中的前 4 行和训练集中的最后一行.先添加一列rank
:
You'd want the first 4 rows in your training set and the last one in your training set. First add a column rank
:
from pyspark.sql.functions import percent_rank
from pyspark.sql import Window
df = df.withColumn("rank", percent_rank().over(Window.partitionBy().orderBy("date")))
现在使用 rank
将您的数据拆分为 train
和 test
:
Now use rank
to split your data into train
and test
:
train_df = df.where("rank <= .8").drop("rank")
train_df.show()
#+---------------------+---+
#|date |x |
#+---------------------+---+
#|2018-01-01 00:00:00.0|0 |
#|2018-01-02 00:00:00.0|1 |
#|2018-01-03 00:00:00.0|2 |
#|2018-01-04 00:00:00.0|3 |
#+---------------------+---+
test_df = df.where("rank > .8").drop("rank")
test_df.show()
#+---------------------+---+
#|date |x |
#+---------------------+---+
#|2018-01-05 00:00:00.0|4 |
#+---------------------+---+
这篇关于将时间序列 pySpark 数据帧拆分为 test &不使用随机拆分训练的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!