AUC较高，但数据不平衡时预测较差 [英] High AUC but bad predictions with imbalanced data

问题描述

我正在尝试在非常不平衡的数据集上使用LightGBM构建分类器.不平衡的比率为97:3，即:

I am trying to build a classifier with LightGBM on a very imbalanced dataset. Imbalance is in the ratio 97:3, i.e.:

Class

0    0.970691
1    0.029309

我使用的参数和训练代码如下所示.

Params I used and the code for training is as shown below.

lgb_params = {
        'boosting_type': 'gbdt',
        'objective': 'binary',
        'metric':'auc',
        'learning_rate': 0.1,
        'is_unbalance': 'true',  #because training data is unbalance (replaced with scale_pos_weight)
        'num_leaves': 31,  # we should let it be smaller than 2^(max_depth)
        'max_depth': 6, # -1 means no limit
        'subsample' : 0.78
    }

# Cross-validate
cv_results = lgb.cv(lgb_params, dtrain, num_boost_round=1500, nfold=10, 
                    verbose_eval=10, early_stopping_rounds=40)

nround = cv_results['auc-mean'].index(np.max(cv_results['auc-mean']))
print(nround)

model = lgb.train(lgb_params, dtrain, num_boost_round=nround)


preds = model.predict(test_feats)

preds = [1 if x >= 0.5 else 0 for x in preds]

我运行简历来获得最好的模型和最好的回合.我在简历上获得了0.994的AUC，并且在验证集中获得了类似的分数.

I ran CV to get the best model and best round. I got 0.994 AUC on CV and similar score in Validation set.

但是当我根据测试集进行预测时，我得到的结果非常糟糕.我敢肯定火车的样本是完美的.

But when I am predicting on the test set I am getting very bad results. I am sure that the train set is sampled perfectly.

需要调整哪些参数?是什么原因造成的?我是否应该对数据集重新采样以减少最高等级??

What parameters are needed to be tuned.? What is the reason for the problem.? Should I resample the dataset such that the highest class is reduced.?

推荐答案

问题是，尽管数据集中的极端类别不平衡，但是在确定

The issue is that, despite the extreme class imbalance in your dataset, you are still using the "default" threshold of 0.5 when deciding the final hard classification in

preds = [1 if x >= 0.5 else 0 for x in preds]

在这种情况下不是.

这是一个相当大的话题，我强烈建议您进行自己的研究(尝试搜索阈值或切断概率不平衡数据)，但是这里有一些指导您入门的指针...

This is a rather big topic, and I strongly suggest you do your own research (try googling for threshold or cut off probability imbalanced data), but here are some pointers to get you started...

来自相关答案，网址为交叉验证(添加了重点):

From a relevant answer at Cross Validated (emphasis added):

别忘了您应该明智地阈值来进行预测.当模型概率大于0.5时，并不总是最好地预测1.另一个阈值可能会更好.为此，您应该查看分类器的接收器工作特征(ROC)曲线，而不仅仅是具有默认概率阈值的预测成功.

Don't forget that you should be thresholding intelligently to make predictions. It is not always best to predict 1 when the model probability is greater 0.5. Another threshold may be better. To this end you should look into the Receiver Operating Characteristic (ROC) curves of your classifier, not just its predictive success with a default probability threshold.

从相关的学术论文中，在不平衡分类中找到最佳分类阈值:

From a relevant academic paper, Finding the Best Classification Threshold in Imbalanced Classification:

2.2.如何设置测试集的分类阈值

预测 结果 是 最终 决定 根据 到 预言 概率. 这 临界点 是 通常 放 到 0.5. 如果 这 预言 可能性 超过 0.5， 这 样本 是 预料到的 到 是 积极的; 否则， 消极的. 然而， 0.5 是 不是 理想的 为了 一些 情况， 特别 为了 不平衡的 数据集.

Prediction results are ultimately determined according to prediction probabilities. The threshold is typically set to 0.5. If the prediction probability exceeds 0.5, the sample is predicted to be positive; otherwise, negative. However, 0.5 is not ideal for some cases, particularly for imbalanced datasets.

帖子(<强烈推荐)中的优化类不平衡的概率阈值 Applied Predictive Modeling博客也与此相关.

The post Optimizing Probability Thresholds for Class Imbalances from the (highly recommended) Applied Predictive Modeling blog is also relevant.

从以上所有内容中吸取教训:AUC很少，但是ROC曲线本身通常是您最好的朋友...

Take home lesson from all the above: AUC is seldom enough, but the ROC curve itself is often your best friend...

在更一般的层面上，关于阈值本身在分类过程中的作用(至少根据我的经验，许多从业者会犯错)，还请检查

On a more general level regarding the role of the threshold itself in the classification process (which, according to my experience at least, many practitioners get wrong), check also the Classification probability threshold thread (and the provided links) at Cross Validated; key point:

当您为新样本的每个类别输出概率时，练习的统计部分结束.选择一个阈值以将新观察值分类为1 vs. 0不再是统计的一部分.它是决定组件的一部分.

the statistical component of your exercise ends when you output a probability for each class of your new sample. Choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component.

这篇关于AUC较高，但数据不平衡时预测较差的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！