如何将数据分成 3 组(训练、验证和测试)? [英] How to split data into 3 sets (train, validation and test)?
问题描述
我有一个 Pandas 数据框,我希望将它分成 3 个独立的集合.我知道使用 sklearn 中的 train_test_split.cross_validation
,可以将数据分成两组(训练和测试).但是,我找不到将数据分成三组的任何解决方案.最好,我想要原始数据的索引.
I have a pandas dataframe and I wish to divide it to 3 separate sets. I know that using train_test_split from sklearn.cross_validation
, one can divide the data in two sets (train and test). However, I couldn't find any solution about splitting the data into three sets. Preferably, I'd like to have the indices of the original data.
我知道一种解决方法是使用 train_test_split
两次并以某种方式调整索引.但是是否有更标准/内置的方法将数据拆分为 3 组而不是 2 组?
I know that a workaround would be to use train_test_split
two times and somehow adjust the indices. But is there a more standard / built-in way to split the data into 3 sets instead of 2?
推荐答案
Numpy 解决方案.我们将首先打乱整个数据集(df.sample(frac=1, random_state=42)
),然后将我们的数据集拆分为以下部分:
Numpy solution. We will shuffle the whole dataset first (df.sample(frac=1, random_state=42)
) and then split our data set into the following parts:
- 60% - 训练集,
- 20% - 验证集,
- 20% - 测试集
In [305]: train, validate, test =
np.split(df.sample(frac=1, random_state=42),
[int(.6*len(df)), int(.8*len(df))])
In [306]: train
Out[306]:
A B C D E
0 0.046919 0.792216 0.206294 0.440346 0.038960
2 0.301010 0.625697 0.604724 0.936968 0.870064
1 0.642237 0.690403 0.813658 0.525379 0.396053
9 0.488484 0.389640 0.599637 0.122919 0.106505
8 0.842717 0.793315 0.554084 0.100361 0.367465
7 0.185214 0.603661 0.217677 0.281780 0.938540
In [307]: validate
Out[307]:
A B C D E
5 0.806176 0.008896 0.362878 0.058903 0.026328
6 0.145777 0.485765 0.589272 0.806329 0.703479
In [308]: test
Out[308]:
A B C D E
4 0.521640 0.332210 0.370177 0.859169 0.401087
3 0.333348 0.964011 0.083498 0.670386 0.169619
[int(.6*len(df)), int(.8*len(df))]
- 是 indices_or_sections 数组"http://docs.scipy.org/doc/numpy/reference/generated/numpy.split.html" rel="noreferrer">numpy.split().
[int(.6*len(df)), int(.8*len(df))]
- is an indices_or_sections
array for numpy.split().
这里是 np.split()
用法的一个小演示 - 让我们将 20 个元素的数组分成以下部分:80%、10%、10%:
Here is a small demo for np.split()
usage - let's split 20-elements array into the following parts: 80%, 10%, 10%:
In [45]: a = np.arange(1, 21)
In [46]: a
Out[46]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
In [47]: np.split(a, [int(.8 * len(a)), int(.9 * len(a))])
Out[47]:
[array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]),
array([17, 18]),
array([19, 20])]
这篇关于如何将数据分成 3 组(训练、验证和测试)?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!