损失与准确性之间的关系 [英] Relationship between loss and accuracy
问题描述
在训练CNN模型时，实际上在每个时期都有可能减少损失并降低准确性吗? 训练时我得到以下结果.
Is it practically possible to have decreasing loss and decreasing accuracy at each epoch when training a CNN model? I am getting the below result while training.
有人可以解释这种情况发生的可能原因吗?
Can someone explain the possible reasons why this is happening?
推荐答案
至少有5个可能导致这种行为的原因:
There are at least 5 reasons which might cause such behavior:
-
异常值::假设您有10张完全相同的图像，其中9张属于 A 类，而其中一个属于 B .在这种情况下，由于大多数示例，模型将开始为该示例分配 A 类的高概率.但是然后-来自异常值的信号可能会破坏模型的稳定性，并使准确性下降.从理论上讲，模型应该稳定在将得分 90％分配给课程 A ，但是它可能会持续很多时期.
Outliers: imagine that you have 10 exactly the same images and 9 out of them belong to class A and one belongs to class B. In this case, a model will start to assign a high probability of class A to this example because of the majority of examples. But then - a signal from outlier might destabilize model and make accuracy decreasing. In theory, a model should stabilize at assigning score 90% to class A but it might last many epochs.
解决方案:为了处理此类示例，我建议您使用渐变剪切(您可以在优化器中添加此类选项).如果要检查是否发生这种现象-您可以检查损失分布(训练集中单个示例的损失)并寻找离群值.
Solutions: In order to deal with such examples I advise you to use gradient clipping (you may add such option in your optimizer). If you want to check if this phenomenon occurs - you may check your losses distributions (losses of individual examples from training set) and look for outliers.
偏差:现在假设您有10张完全相同的图像，但其中5张已分配了 A 类和5个 B .在这种情况下，模型将尝试在这两个类别上分配大约 50％-50％分布.现在-您的模型最多可以达到50％的准确性-从两个有效的类别中选择一个类别.
Bias: Now imagine that you have 10 exactly the same images but 5 of them have assigned class A and 5 - class B. In this case, a model will try to assign approximately 50%-50% distribution on both of these classes. Now - your model can achieve at most 50% of accuracy here - choosing one class out of two valid.
解决方案::尝试增加模型容量-很多时候，您会得到一组非常相似的图像-增加表达能力可能有助于区分相似的示例.不过要当心.另一个解决方案是在训练中尝试此策略.如果要检查是否发生这种现象，请检查各个示例的损失分布.如果分配偏向更高的值-您可能正遭受偏差.
Solution: Try to increase the model capacity - very often you have a set of really similar images - adding expressive power might help to discriminate similar examples. Beware of overfitting though. Another solution is to try this strategy in your training. If you want to check if such phenomenon occurs - check the distribution of losses of individual examples. If a distribution would be skewed toward higher values - you are probably suffering from bias.
类不平衡:现在，假设您的图像中的 90％属于类 A .在培训的早期阶段，您的模型主要集中于将此类分配给几乎所有示例.这可能会使个人损失达到很高的价值，并通过使预测的分布更加不稳定来破坏模型的稳定性.
Class inbalance: Now imagine that 90% of your images belong to class A. In an early stage of your training, your model is mainly concentrating on assigning this class to almost all of examples. This might make individual losses to achieve really high values and destabilize your model by making a predicted distribution more unstable.
解决方案:再次-渐变裁剪.第二件事-耐心，尝试简单地让模型留出更多的时代.模型应该在进一步的训练中学习更多的细节.当然-通过分配sample_weights或class_weights尝试类平衡.如果要检查是否发生这种现象，请检查您的班级分布.
Solution: once again - gradient clipping. Second thing - patience, try simply leaving your model for more epochs. A model should learn more subtle in a further phase of training. And of course - try class balancing - by either assigning sample_weights or class_weights. If you want to check if this phenomenon occurs - check your class distribution.
正则化太强::如果您将正则化设置得过于严格-培训过程主要集中在使权重比实际学习有趣的见解更小的规范上.
Too strong regularization: if you set your regularization to be too strict - a training process is mainly concentrated on making your weights to have smaller norm than actually learning interesting insights.
解决方案:添加categorical_crossentropy作为指标，并观察其是否也在减少.如果不是，则表示您的正则化过于严格，然后尝试分配较少的权重损失.
Solution: add a categorical_crossentropy as a metric and observe if it's also decreasing. If not - then it means that your regularization is too strict - try to assign less weight penalty then.
错误的模型设计-这种行为可能是由错误的模型设计引起的.为了改善您的模型，可以采用几种良好的做法:
Bad model design - such behavior might be caused by a wrong model design. There are several good practices which one might apply in order to improve your model:
批处理规范化-由于采用了这种技术，您可以防止模型因内部网络激活而发生根本变化.这使培训更加稳定和有效.如果批处理量较小，这也许也是对模型进行规范化的一种真正方法.
Batch Normalization - thanks to this technique you are preventing your model from radical changes of inner network activations. This makes training much more stable and efficient. With a small batch size, this might be also a genuine way of regularizing your model.
渐变裁剪-这使您的模型训练更加稳定和有效.
Gradient clipping - this makes your model training much more stable and efficient.
减少瓶颈效应-阅读这篇很棒的论文，然后检查您的模型可能会遇到瓶颈问题.
Reduce bottleneck effect - read this fantastic paper and check if your model might suffer from bottleneck problem.
添加辅助分类器-如果您要从头开始训练网络-这将使您的功能更有意义，并且训练-更快，更有效.
Add auxiliary classifiers - if you are training your network from scratch - this should make your features much more meaningful and your training - faster and more efficient.
这篇关于损失与准确性之间的关系的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
