如何将数据集分割/划分为训练和测试数据集，例如进行交叉验证? [英] How to split/partition a dataset into training and test datasets for, e.g., cross validation?

问题描述

将NumPy数组随机分为训练和测试/验证数据集的好方法是什么?与Matlab中的cvpartition或crossvalind函数类似.

What is a good way to split a NumPy array randomly into training and testing/validation dataset? Something similar to the cvpartition or crossvalind functions in Matlab.

推荐答案

如果要将数据集分成两半，可以使用numpy.random.shuffle或numpy.random.permutation(如果需要跟踪索引) :

If you want to split the data set once in two halves, you can use numpy.random.shuffle, or numpy.random.permutation if you need to keep track of the indices:

import numpy
# x is your dataset
x = numpy.random.rand(100, 5)
numpy.random.shuffle(x)
training, test = x[:80,:], x[80:,:]

或

import numpy
# x is your dataset
x = numpy.random.rand(100, 5)
indices = numpy.random.permutation(x.shape[0])
training_idx, test_idx = indices[:80], indices[80:]
training, test = x[training_idx,:], x[test_idx,:]

有许多方法重复分割同一数据集以进行交叉验证.一种策略是从数据集中重复采样:

There are many ways to repeatedly partition the same data set for cross validation. One strategy is to resample from the dataset, with repetition:

import numpy
# x is your dataset
x = numpy.random.rand(100, 5)
training_idx = numpy.random.randint(x.shape[0], size=80)
test_idx = numpy.random.randint(x.shape[0], size=20)
training, test = x[training_idx,:], x[test_idx,:]

最后， sklearn 包含方法可创建相对于某些功能平衡的数据分区，例如，确保训练和测试集中的正例和负例比例相同.

Finally, sklearn contains several cross validation methods (k-fold, leave-n-out, ...). It also includes more advanced "stratified sampling" methods that create a partition of the data that is balanced with respect to some features, for example to make sure that there is the same proportion of positive and negative examples in the training and test set.

这篇关于如何将数据集分割/划分为训练和测试数据集，例如进行交叉验证?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！