scikit-learn 中每个数据拆分的交叉验证指标 [英] Cross-validation metrics in scikit-learn for each data split

问题描述

我需要为 (X_test, y_test) 数据的每个拆分明确获取交叉验证统计数据.

I need to get the cross-validation statistics explicitly for each split of the (X_test, y_test) data.

所以，为了尝试这样做，我做到了:

So, to try to do so I did:

kf = KFold(n_splits=n_splits)

X_train_tmp = []
y_train_tmp = []
X_test_tmp = []
y_test_tmp = []
mae_train_cv_list = []
mae_test_cv_list = []

for train_index, test_index in kf.split(X_train):
    
    for i in range(len(train_index)):
        X_train_tmp.append(X_train[train_index[i]])
        y_train_tmp.append(y_train[train_index[i]])

    for i in range(len(test_index)):
        X_test_tmp.append(X_train[test_index[i]])
        y_test_tmp.append(y_train[test_index[i]])

    model.fit(X_train_tmp, y_train_tmp) # FIT the model = SVR, NN, etc.

    mae_train_cv_list.append( mean_absolute_error(y_train_tmp, model.predict(X_train_tmp)) # MAE of the train part of the KFold.

    mae_test_cv_list.append( mean_absolute_error(y_test_tmp, model.predict(X_test_tmp)) ) # MAE of the test part of the KFold.

    X_train_tmp = []
    y_train_tmp = []
    X_test_tmp = []
    y_test_tmp = []

这是使用 KFold 等获得每个交叉验证拆分的平均绝对误差 (MAE) 的正确方法吗?

Is it the proper way of getting the Mean Absolute Error (MAE) for each cross-validation split by using, for instance, KFold?

推荐答案

您的方法存在一些问题.

There are some issues with your approach.

首先，您当然不必在训练中手动地附加数据&验证列表(即您的 2 个内部 for 循环)；简单的索引就可以完成这项工作.

To start with, you certainly don't have to append the data manually one by one in your training & validation lists (i.e. your 2 inner for loops); simple indexing will do the job.

此外，我们通常从不计算 &报告训练 CV 折叠的错误 - 仅报告验证折叠的错误.

Additionally, we normally never compute & report the error of the training CV folds - only the error on the validation folds.

牢记这些，并将术语切换为验证"而不是测试"，这是一个使用波士顿数据的简单可重现示例，它应该可以直接适应您的情况:

Keeping these in mind, and switching the terminology to "validation" instead of "test", here is a simple reproducible example using the Boston data, which should be straighforward to adapt to your case:

from sklearn.model_selection import KFold
from sklearn.datasets import load_boston
from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeRegressor

X, y = load_boston(return_X_y=True)
n_splits = 5
kf = KFold(n_splits=n_splits, shuffle=True)
model = DecisionTreeRegressor(criterion='mae')

cv_mae = []

for train_index, val_index in kf.split(X):
    model.fit(X[train_index], y[train_index])
    pred = model.predict(X[val_index])
    err = mean_absolute_error(y[val_index], pred)
    cv_mae.append(err)

之后，你的cv_mae应该是这样的(由于简历的随机性，细节会有所不同):

after which, your cv_mae should be something like (details will differ due to the random nature of CV):

[3.5294117647058827,
 3.3039603960396042,
 3.5306930693069307,
 2.6910891089108913,
 3.0663366336633664]

当然，所有这些明确的东西并不是真正必要的；你可以更简单地使用 cross_val_score.不过有一个小问题:

Of course, all this explicit stuff is not really necessary; you could do the job much more simply with cross_val_score. There is a small catch though:

from sklearn.model_selection import cross_val_score
cv_mae2 =cross_val_score(model, X, y, cv=n_splits, scoring="neg_mean_absolute_error")
cv_mae2
# result
array([-2.94019608, -3.71980198, -4.92673267, -4.5990099 , -4.22574257])

除了实际上不是问题的负号之外，您会注意到结果的方差与上面的 cv_mae 相比看起来要高得多；原因是我们没有洗牌我们的数据.不幸的是，cross_val_score 没有提供 shuffle 选项，所以我们必须使用 shuffle 手动完成.所以我们最终的代码应该是:

Apart from the negative sign which is not really an issue, you'll notice that the variance of the results looks significantly higher compared to our cv_mae above; and the reason is that we didn't shuffle our data. Unfortunately, cross_val_score does not provide a shuffling option, so we have to do this manually using shuffle. So our final code should be:

from sklearn.model_selection import cross_val_score
from sklearn.utils import shuffle
X_s, y_s =shuffle(X, y)
cv_mae3 =cross_val_score(model, X_s, y_s, cv=n_splits, scoring="neg_mean_absolute_error")
cv_mae3
# result:
array([-3.24117647, -3.57029703, -3.10891089, -3.45940594, -2.78316832])

折叠之间的差异明显更小，并且更接近我们最初的cv_mae...

which is of significantly less variance between the folds, and much closer to our initial cv_mae...

这篇关于scikit-learn 中每个数据拆分的交叉验证指标的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！