在 sklearn cross_val_score 上评估多个分数 [英] Evaluate multiple scores on sklearn cross_val_score
问题描述
我正在尝试使用 sklearn 来评估多个机器学习算法的几个指标(准确度、召回率、精确度等等).
对于我从文档这里和源代码中了解到的内容(我使用 sklearn 0.17)，cross_val_score 函数每次执行只接收一个得分手.所以为了计算多个分数，我必须:
- 多次执行
实现我的(耗时且容易出错的)评分器
我用这段代码执行了多次:
from sklearn.svm import SVC从 sklearn.naive_bayes 导入 GaussianNB从 sklearn.tree 导入 DecisionTreeClassifier从 sklearn.cross_validation 导入 cross_val_score导入时间从 sklearn.datasets 导入 load_iris虹膜 = load_iris()模型 = [GaussianNB(), DecisionTreeClassifier(), SVC()]名称 = [朴素贝叶斯"、决策树"、SVM"]对于模型，zip 中的名称(模型，名称):印刷名称开始 = time.time()对于 ["accuracy", "precision", "recall"] 中的分数:打印分数，打印 " : "，打印 cross_val_score(model, iris.data, iris.target,scoring=score, cv=10).mean()打印 time.time() - 开始
我得到这个输出:
朴素贝叶斯精度:0.9533333333333精度:0.962698412698召回:0.95333333333330.0383198261261决策树精度:0.9533333333333精度:0.9588888888889召回:0.95333333333330.0494720935822支持向量机准确度:0.98精度:0.983333333333回忆:0.980.063080072403没关系，但我自己的数据很慢.如何衡量所有分数?
自从写这篇文章以来，scikit-learn 已经更新并且我的答案已经过时，请参阅下面的更简洁的解决方案><小时>
您可以编写自己的评分函数来捕获所有三个信息，但是用于交叉验证的评分函数必须只返回 scikit-learn 中的一个数字(这很可能用于兼容性原因).下面是一个示例，其中每个交叉验证切片的每个分数都打印到控制台，并且返回值只是三个指标的总和.如果要返回所有这些值，则必须对 cross_val_score(cross_validation.py 的第 1351 行)和 _score(第 1601 行或相同的文件).
from sklearn.svm import SVC从 sklearn.naive_bayes 导入 GaussianNB从 sklearn.tree 导入 DecisionTreeClassifier从 sklearn.cross_validation 导入 cross_val_score导入时间从 sklearn.datasets 导入 load_iris从 sklearn.metrics 导入accuracy_score、precision_score、recall_score虹膜 = load_iris()模型 = [GaussianNB(), DecisionTreeClassifier(), SVC()]名称 = [朴素贝叶斯"、决策树"、SVM"]def getScores(estimator, x, y):yPred = estimator.predict(x)返回 (accuracy_score(y, yPred),precision_score(y, yPred, pos_label=3,average='macro'),召回分数(y，yPred，pos_label=3，average='macro'))def my_scorer(estimator, x, y):a, p, r = getScores(estimator, x, y)打印 a, p, r返回 a+p+r对于模型，zip 中的名称(模型，名称):印刷名称开始 = time.time()m = cross_val_score(model, iris.data, iris.target,scoring=my_scorer, cv=10).mean()打印 '
Sum:',m, '
'打印 'time', time.time() - 开始，'
'哪个给出:
朴素贝叶斯0.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333331.0 1.0 1.00.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333330.866666666667 0.904761904762 0.8666666666671.0 1.0 1.01.0 1.0 1.01.0 1.0 1.0总和:2.86936507937时间 0.0249638557434决策树1.0 1.0 1.00.933333333333 0.944444444444 0.9333333333331.0 1.0 1.00.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333330.866666666667 0.866666666667 0.8666666666670.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333331.0 1.0 1.01.0 1.0 1.0总和:2.86555555556时间 0.0237860679626支持向量机1.0 1.0 1.00.933333333333 0.944444444444 0.9333333333331.0 1.0 1.01.0 1.0 1.01.0 1.0 1.00.933333333333 0.944444444444 0.9333333333330.933333333333 0.944444444444 0.9333333333331.0 1.0 1.01.0 1.0 1.01.0 1.0 1.0总和:2.94333333333时间 0.043044090271<小时>
从 scikit-learn 0.19.0 开始，解决方案变得更容易
from sklearn.model_selection import cross_validate从 sklearn.datasets 导入 load_iris从 sklearn.svm 导入 SVC虹膜 = load_iris()clf = SVC()评分 = {'acc':'准确度'，'prec_macro': 'precision_macro','rec_micro': 'recall_macro'}分数 = cross_validate(clf, iris.data, iris.target, score=scoring,cv=5，return_train_score=真)打印(scores.keys())打印(分数['test_acc'])给出:
['test_acc', 'score_time', 'train_acc', 'fit_time', 'test_rec_micro', 'train_rec_micro', 'train_prec_macro', 'test_prec_macro'][ 0.96666667 1. 0.96666667 0.96666667 1. ]I'm trying to evaluate multiple machine learning algorithms with sklearn for a couple of metrics (accuracy, recall, precision and maybe more).
For what I understood from the documentation here and from the source code(I'm using sklearn 0.17), the cross_val_score function only receives one scorer for each execution. So for calculating multiple scores, I have to :
- Execute multiple times
Implement my (time consuming and error prone) scorer
I've executed multiple times with this code :
from sklearn.svm import SVC from sklearn.naive_bayes import GaussianNB from sklearn.tree import DecisionTreeClassifier from sklearn.cross_validation import cross_val_score import time from sklearn.datasets import load_iris iris = load_iris() models = [GaussianNB(), DecisionTreeClassifier(), SVC()] names = ["Naive Bayes", "Decision Tree", "SVM"] for model, name in zip(models, names): print name start = time.time() for score in ["accuracy", "precision", "recall"]: print score, print " : ", print cross_val_score(model, iris.data, iris.target,scoring=score, cv=10).mean() print time.time() - start
And I get this output:
Naive Bayes
accuracy : 0.953333333333
precision : 0.962698412698
recall : 0.953333333333
0.0383198261261
Decision Tree
accuracy : 0.953333333333
precision : 0.958888888889
recall : 0.953333333333
0.0494720935822
SVM
accuracy : 0.98
precision : 0.983333333333
recall : 0.98
0.063080072403
Which is ok, but it's slow for my own data. How can I measure all scores ?
Since the time of writing this post scikit-learn has updated and made my answer obsolete, see the much cleaner solution below
You can write your own scoring function to capture all three pieces of information, however a scoring function for cross validation must only return a single number in scikit-learn (this is likely for compatibility reasons). Below is an example where each of the scores for each cross validation slice prints to the console, and the returned value is just the sum of the three metrics. If you want to return all these values, you're going to have to make some changes to cross_val_score (line 1351 of cross_validation.py) and _score (line 1601 or the same file).
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.cross_validation import cross_val_score
import time
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score, precision_score, recall_score
iris = load_iris()
models = [GaussianNB(), DecisionTreeClassifier(), SVC()]
names = ["Naive Bayes", "Decision Tree", "SVM"]
def getScores(estimator, x, y):
yPred = estimator.predict(x)
return (accuracy_score(y, yPred),
precision_score(y, yPred, pos_label=3, average='macro'),
recall_score(y, yPred, pos_label=3, average='macro'))
def my_scorer(estimator, x, y):
a, p, r = getScores(estimator, x, y)
print a, p, r
return a+p+r
for model, name in zip(models, names):
print name
start = time.time()
m = cross_val_score(model, iris.data, iris.target,scoring=my_scorer, cv=10).mean()
print '
Sum:',m, '
'
print 'time', time.time() - start, '
'
Which gives:
Naive Bayes
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
1.0 1.0 1.0
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
0.866666666667 0.904761904762 0.866666666667
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
Sum: 2.86936507937
time 0.0249638557434
Decision Tree
1.0 1.0 1.0
0.933333333333 0.944444444444 0.933333333333
1.0 1.0 1.0
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
0.866666666667 0.866666666667 0.866666666667
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
1.0 1.0 1.0
1.0 1.0 1.0
Sum: 2.86555555556
time 0.0237860679626
SVM
1.0 1.0 1.0
0.933333333333 0.944444444444 0.933333333333
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
0.933333333333 0.944444444444 0.933333333333
0.933333333333 0.944444444444 0.933333333333
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0
Sum: 2.94333333333
time 0.043044090271
As of scikit-learn 0.19.0 the solution becomes much easier
from sklearn.model_selection import cross_validate
from sklearn.datasets import load_iris
from sklearn.svm import SVC
iris = load_iris()
clf = SVC()
scoring = {'acc': 'accuracy',
'prec_macro': 'precision_macro',
'rec_micro': 'recall_macro'}
scores = cross_validate(clf, iris.data, iris.target, scoring=scoring,
cv=5, return_train_score=True)
print(scores.keys())
print(scores['test_acc'])
Which gives:
['test_acc', 'score_time', 'train_acc', 'fit_time', 'test_rec_micro', 'train_rec_micro', 'train_prec_macro', 'test_prec_macro']
[ 0.96666667 1. 0.96666667 0.96666667 1. ]
这篇关于在 sklearn cross_val_score 上评估多个分数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
