Scikit学习:在GridSearchCV中评分 [英] Scikit-learn: scoring in GridSearchCV

问题描述

似乎scikit-learn的GridSearchCV收集了其(内部)交叉验证折叠的得分，然后平均所有折叠的得分.我想知道背后的理由.乍一看，似乎更灵活地是收集交叉验证折叠的预测，然后将选定的评分标准应用于所有折叠的预测.

It seems that GridSearchCV of scikit-learn collects the scores of its (inner) cross-validation folds and then averages across the scores of all folds. I was wondering about the rationale behind this. At first glance, it would seem more flexible to instead collect the predictions of its cross-validation folds and then apply the chosen scoring metric to the predictions of all folds.

我偶然发现的原因是我在cv=LeaveOneOut()和scoring='balanced_accuracy'(scikit-learn v0.20.dev0)的不平衡数据集上使用了GridSearchCV.对每个遗漏的样本应用评分指标(例如平衡的准确性(或查全率))是没有意义的.相反，我想先收集所有预测，然后将我的得分指标一次应用于所有预测.还是这涉及推理错误?

The reason I stumbled upon this is that I use GridSearchCV on an imbalanced data set with cv=LeaveOneOut() and scoring='balanced_accuracy' (scikit-learn v0.20.dev0). It doesn't make sense to apply a scoring metric such as balanced accuracy (or recall) to each left-out sample. Rather, I would want to collect all predictions first and then apply my scoring metric once to all predictions. Or does this involve an error in reasoning?

更新:我通过基于GridSearchCV创建自定义网格搜索类来解决此问题，区别在于首先从所有内部折叠中收集预测，并且一次应用评分标准.

Update: I solved it by creating a custom grid search class based on GridSearchCV with the difference that predictions are first collected from all inner folds and the scoring metric is applied once.

推荐答案

GridSearchCV使用评分来决定在模型中设置哪些内部超参数.

GridSearchCVuses the scoring to decide what internal hyperparameters to set in the model.

如果要估计最佳"超参数的性能，则需要执行交叉验证的附加步骤.

If you want to estimate the performance of the "optimal" hyperparameters, you need to do an additional step of cross validation.

请参见 http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html

编辑以更接近回答实际问题: 对我来说，如果您想使用LeaveOneOut和balanced_accuracy，则收集每个折叠的预测然后对它们进行评分似乎是合理的.我想您需要创建自己的网格搜索器才能做到这一点.您可以为此使用model_selection.ParameterGrid和model_selection.KFold.

EDIT to get closer to answering the actual question: For me it seems reasonable to collect predictions for each fold and then score them all, if you want to use LeaveOneOut and balanced_accuracy. I guess you need to make your own grid searcher to do that. You could use model_selection.ParameterGrid and model_selection.KFold for that.

这篇关于Scikit学习:在GridSearchCV中评分的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！