scikit-learn cross_val_predict 准确率分数是如何计算的? [英] How is scikit-learn cross_val_predict accuracy score calculated?

问题描述

是否cross_val_predict(参见doc, v0.18) 使用 k-fold 方法，如下面的代码所示计算每个折叠的准确度并最终对它们求平均值?

cv = KFold(len(labels), n_folds=20)clf = SVC()ypred = cross_val_predict(clf, td, 标签, cv=cv)准确度 = 准确度_分数(标签，ypred)印刷精度

解决方案

不，它没有！

根据交叉验证文档页面，cross_val_predict 不返回任何分数，而只返回基于此处描述的特定策略的标签:

<块引用>

cross_val_predict 函数有一个类似的接口cross_val_score, 但返回，对于输入中的每个元素，在测试中为该元素获得的预测设置.只有将所有元素分配给一个的交叉验证策略测试集只能使用一次(否则会引发异常).

因此，通过调用 accuracy_score(labels, ypred) 您只是在计算上述特定策略预测的标签与真实标签相比的准确度分数.这在同一文档页面中再次指定:

<块引用>

然后可以使用这些预测来评估分类器:

predicted = cross_val_predict(clf, iris.data, iris.target, cv=10)metrics.accuracy_score(iris.target，预测)

注意这个计算的结果可能会略有不同从使用 cross_val_score 作为元素分组获得的那些以不同的方式.

如果您需要不同折叠的准确度分数，您应该尝试:

<预><代码>>>>分数 = cross_val_score(clf, X, y, cv=cv)>>>分数数组([ 0.96..., 1. ..., 0.96..., 0.96..., 1. ])

然后对于所有折叠的平均准确度使用 scores.mean():

<预><代码>>>>打印(准确度:%0.2f(+/- %0.2f)"%(scores.mean()，scores.std()* 2))准确度:0.98 (+/- 0.03)

<小时>

如何计算每个折叠的 Cohen kappa 系数和混淆矩阵?

为了计算 Cohen Kappa 系数 和混淆矩阵，我假设你的意思是真实标签和每个折叠的预测标签之间的 kappa 系数和混淆矩阵:

from sklearn.model_selection import KFold从 sklearn.svm.classes 导入 SVC从 sklearn.metrics.classification 导入 cohen_kappa_score从 sklearn.metrics 导入混淆_矩阵cv = KFold(len(labels), n_folds=20)clf = SVC()对于 cv.split(X) 中的 train_index、test_index:clf.fit(X[train_index], 标签[train_index])ypred = clf.predict(X[test_index])kappa_score = cohen_kappa_score(标签[test_index]，ypred)混淆矩阵=混淆矩阵(标签[测试索引]，ypred)

<小时>

cross_val_predict 返回什么?

它使用 KFold 将数据拆分为 k 部分，然后进行 i=1..k 迭代:

• 取i'th部分作为测试数据，其他部分作为训练数据
• 使用训练数据训练模型(除 i'th 之外的所有部分)
• 然后通过使用这个训练好的模型，预测 i'th 部分(测试数据)的标签

在每次迭代中，预测数据的 i'th 部分的标签.最后cross_val_predict合并所有部分预测的标签，作为最终结果返回.

这段代码一步一步地展示了这个过程:

X = np.array([[0], [1], [2], [3], [4], [5]])标签 = np.array(['a', 'a', 'a', 'b', 'b', 'b'])cv = KFold(len(labels), n_folds=3)clf = SVC()ypred_all = np.chararray((labels.shape))我 = 1对于 cv.split(X) 中的 train_index、test_index:打印(迭代"，我，:")打印(火车索引:"，train_index)打印(火车数据:"，X[train_index])打印(测试索引:"，test_index)打印(测试数据:"，X[test_index])clf.fit(X[train_index], 标签[train_index])ypred = clf.predict(X[test_index])打印(索引数据的预测标签"，test_index，是:"，ypred)ypred_all[test_index] = ypred打印(合并预测标签:"，ypred_all)我=我+1打印(====================================)y_cross_val_predict = cross_val_predict(clf, X, 标签, cv=cv)print("cross_val_predict 预测的标签:", y_cross_val_predict)

结果是:

迭代1:火车指数:[2 3 4 5]训练数据:[[2] [3] [4] [5]]测试指标:[0 1]测试数据:[[0] [1]]索引 [0 1] 数据的预测标签是:['b' 'b']合并预测标签:['b' 'b' '' '' '' '']======================================迭代2:火车指数:[0 1 4 5]训练数据:[[0] [1] [4] [5]]测试指标:[2 3]测试数据:[[2] [3]]索引 [2 3] 数据的预测标签是:['a' 'b']合并预测标签:['b' 'b' 'a' 'b' '' '']======================================迭代3:火车指数:[0 1 2 3]训练数据:[[0] [1] [2] [3]]测试指标:[4 5]测试数据:[[4] [5]]索引 [4 5] 数据的预测标签是: ['a' 'a']合并预测标签:['b' 'b' 'a' 'b' 'a' 'a']======================================cross_val_predict 预测的标签:['b' 'b' 'a' 'b' 'a' 'a']

Does the cross_val_predict (see doc, v0.18) with k-fold method as shown in the code below calculate accuracy for each fold and average them finally or not?

cv = KFold(len(labels), n_folds=20)
clf = SVC()
ypred = cross_val_predict(clf, td, labels, cv=cv)
accuracy = accuracy_score(labels, ypred)
print accuracy

解决方案

No, it does not!

According to cross validation doc page, cross_val_predict does not return any scores but only the labels based on a certain strategy which is described here:

The function cross_val_predict has a similar interface to cross_val_score, but returns, for each element in the input, the prediction that was obtained for that element when it was in the test set. Only cross-validation strategies that assign all elements to a test set exactly once can be used (otherwise, an exception is raised).

And therefore by calling accuracy_score(labels, ypred) you are just calculating accuracy scores of labels predicted by aforementioned particular strategy compared to the true labels. This again is specified in the same documentation page:

These prediction can then be used to evaluate the classifier:

predicted = cross_val_predict(clf, iris.data, iris.target, cv=10) 
metrics.accuracy_score(iris.target, predicted)

Note that the result of this computation may be slightly different from those obtained using cross_val_score as the elements are grouped in different ways.

If you need accuracy scores of different folds you should try:

>>> scores = cross_val_score(clf, X, y, cv=cv)
>>> scores                                              
array([ 0.96...,  1.  ...,  0.96...,  0.96...,  1.        ])

and then for the mean accuracy of all folds use scores.mean():

>>> print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
Accuracy: 0.98 (+/- 0.03)

---

How to calculate Cohen kappa coefficient and confusion matrix for each fold?

For calculating Cohen Kappa coefficient and confusion matrix I assumed you mean kappa coefficient and confusion matrix between true labels and each fold's predicted labels:

from sklearn.model_selection import KFold
from sklearn.svm.classes import SVC
from sklearn.metrics.classification import cohen_kappa_score
from sklearn.metrics import confusion_matrix

cv = KFold(len(labels), n_folds=20)
clf = SVC()
for train_index, test_index in cv.split(X):
    clf.fit(X[train_index], labels[train_index])
    ypred = clf.predict(X[test_index])
    kappa_score = cohen_kappa_score(labels[test_index], ypred)
    confusion_matrix = confusion_matrix(labels[test_index], ypred)

---

What does cross_val_predict return?

It uses KFold to split the data to k parts and then for i=1..k iterations:

• takes i'th part as the test data and all other parts as training data
• trains the model with training data (all parts except i'th)
• then by using this trained model, predicts labels for i'th part (test data)

In each iteration, label of i'th part of data gets predicted. In the end cross_val_predict merges all partially predicted labels and returns them as the final result.

This code shows this process step by step:

X = np.array([[0], [1], [2], [3], [4], [5]])
labels = np.array(['a', 'a', 'a', 'b', 'b', 'b'])

cv = KFold(len(labels), n_folds=3)
clf = SVC()
ypred_all = np.chararray((labels.shape))
i = 1
for train_index, test_index in cv.split(X):
    print("iteration", i, ":")
    print("train indices:", train_index)
    print("train data:", X[train_index])
    print("test indices:", test_index)
    print("test data:", X[test_index])
    clf.fit(X[train_index], labels[train_index])
    ypred = clf.predict(X[test_index])
    print("predicted labels for data of indices", test_index, "are:", ypred)
    ypred_all[test_index] = ypred
    print("merged predicted labels:", ypred_all)
    i = i+1
    print("=====================================")
y_cross_val_predict = cross_val_predict(clf, X, labels, cv=cv)
print("predicted labels by cross_val_predict:", y_cross_val_predict)

The result is:

iteration 1 :
train indices: [2 3 4 5]
train data: [[2] [3] [4] [5]]
test indices: [0 1]
test data: [[0] [1]]
predicted labels for data of indices [0 1] are: ['b' 'b']
merged predicted labels: ['b' 'b' '' '' '' '']
=====================================
iteration 2 :
train indices: [0 1 4 5]
train data: [[0] [1] [4] [5]]
test indices: [2 3]
test data: [[2] [3]]
predicted labels for data of indices [2 3] are: ['a' 'b']
merged predicted labels: ['b' 'b' 'a' 'b' '' '']
=====================================
iteration 3 :
train indices: [0 1 2 3]
train data: [[0] [1] [2] [3]]
test indices: [4 5]
test data: [[4] [5]]
predicted labels for data of indices [4 5] are: ['a' 'a']
merged predicted labels: ['b' 'b' 'a' 'b' 'a' 'a']
=====================================
predicted labels by cross_val_predict: ['b' 'b' 'a' 'b' 'a' 'a']

这篇关于scikit-learn cross_val_predict 准确率分数是如何计算的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！