使用多个分类器时-如何衡量整体表现? [SciKit学习] [英] When using multiple classifiers - How to measure the ensemble's performance? [SciKit Learn]

问题描述

我有一个分类问题(预测一个序列是否属于一个类)，为此，我决定使用多种分类方法，以帮助滤除误报.

I have a classification problem (predicting whether a sequence belongs to a class or not), for which I decided to use multiple classification methods, in order to help filter out the false positives.

(问题出在生物信息学中-将蛋白质序列归类为神经肽前体序列.如果有兴趣的人，这里是原始文章，以及用于生成特征和训练单个预测变量的代码).

(The problem is in bioinformatics - classifying protein sequences as being Neuropeptide precursors sequences. Here's the original article if anyone's interested, and the code used to generate features and to train a single predictor) .

现在，分类器具有大致相似的性能指标(针对10折CV的训练集，其准确度/准确性/准确性为83-94％)，所以我的天真"方法是简单地使用多个分类器(Random Forests， ExtraTrees，SVM(线性内核)，SVM(RBF内核)和GRB)，并使用简单的多数表决.

Now, the classifiers have roughly similar performance metrics (83-94% accuracy/precision/etc' on the training set for 10-fold CV), so my 'naive' approach was to simply use multiple classifiers (Random Forests, ExtraTrees, SVM (Linear kernel), SVM (RBF kernel) and GRB) , and to use a simple majority vote.

我的问题是: 如何获得不同分类器和/或其投票预测的性能指标? 也就是说，我想看看使用多个分类器是否可以根本改善我的性能，或者使用它们的组合.

MY question is: How can I get the performance metrics for the different classifiers and/or their votes predictions? That is, I want to see if using the multiple classifiers improves my performance at all, or which combination of them does.

我的直觉可能是使用ROC得分，但我不知道如何组合"结果并从分类器的组合中获得结果. (即，仅查看每个分类器的ROC曲线[已经知道]，然后使用分类器的组合查看训练数据的ROC曲线或AUC).

My intuition is maybe to use the ROC score, but I don't know how to "combine" the results and to get it from a combination of classifiers. (That is, to see what the ROC curve is just for each classifier alone [already known], then to see the ROC curve or AUC for the training data using combinations of classifiers).

(我目前使用Random Forests和ExtraTrees方法使用预测概率"过滤预测，然后针对预测得分低于'0.85'的结果任意过滤.另一层过滤是有多少个分类器对此达成共识蛋白的阳性分类").

(I currently filter the predictions using "predict probabilities" with the Random Forests and ExtraTrees methods, then I filter arbitrarily for results with a predicted score below '0.85'. An additional layer of filtering is "how many classifiers agree on this protein's positive classification").

非常感谢您！

(网站实现，我们在其中使用多个分类器- http://neuropid.cs.huji.ac.il/)

(The website implementation, where we're using the multiple classifiers - http://neuropid.cs.huji.ac.il/ )

整个shebang是使用SciKit学习和python实现的.引用和全部！)

The whole shebang is implemented using SciKit learn and python. Citations and all!)

推荐答案

要评估集成的性能，只需遵循与通常相同的方法即可.但是，您将需要首先获得10折数据集分区，然后针对每折，在同一折上训练所有合奏，测量准确性，冲洗并重复其他折，然后计算出合奏的准确性.因此，关键的区别是在评估整体时不使用k折交叉验证来训练各个算法.重要的是不要让集合直接看到测试数据，也不要让其中一种算法看到测试数据.

To evaluate the performance of the ensemble, simply follow the same approach as you would normally. However, you will want to get the 10 fold data set partitions first, and for each fold, train all of your ensemble on that same fold, measure the accuracy, rinse and repeat with the other folds and then compute the accuracy of the ensemble. So the key difference is to not train the individual algorithms using k fold cross-validation when evaluating the ensemble. The important thing is not to let the ensemble see the test data either directly or by letting one of it's algorithms see the test data.

还请注意，RF和Extra Trees本身已经是集成算法.

Note also that RF and Extra Trees are already ensemble algorithms in their own right.

另一种方法(再次确保整体方法)是采用分类器输出的概率和\或标签，并将它们输入另一个产生预测的分类器(例如DT，RF，SVM等)通过结合来自其他分类器的最佳猜测.这称为堆叠"

An alternative approach (again making sure the ensemble approach) is to take the probabilities and \ or labels output by your classifiers, and feed them into another classifier (say a DT, RF, SVM, or whatever) that produces a prediction by combining the best guesses from these other classifiers. This is termed "Stacking"

这篇关于使用多个分类器时-如何衡量整体表现? [SciKit学习]的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！