理解COCO评估“最大检测" [英] Understanding COCO evaluation "maximum detections"

问题描述

我开始使用 cocoapi 来评估使用 对象检测 API.在阅读了解释平均精度 (mAP) 和召回率的各种来源后，我对 cocoapi 中使用的最大检测"参数感到困惑.

I started using the cocoapi to evaluate a model trained using the Object Detection API. After reading various sources that explain mean average precision (mAP) and recall, I am confused with the "maximum detections" paramter used in the cocoapi.

据我所知(例如 此处，此处 或这里)，通过计算各种模型分数阈值的精度和召回率来计算 mAP.这给出了精确召回曲线，并且 mAP 被计算为该曲线下面积的近似值.或者，以不同的方式表示为定义的召回范围 (0:0.1:1) 中最大精度的平均值.

From what I understood (e.g. here, here or here), one calculates mAP by calculating precision and recall for various model score thresholds. This gives the precision-recall curve and mAP is calculated as an approximation to the area under this curve. Or, expressed differently, as the average of the maximum precision in defined recall ranges (0:0.1:1).

然而，cocoapi 似乎计算给定数量的最大检测(maxDet)的精度和召回率最高的分数.从那里得到 maxDets = 1, 10, 100 的精确召回曲线.为什么这是一个很好的指标，因为它显然与上述方法不同(它可能不包括数据点)?

However, the cocoapi seems to calculate precision and recall for a given number of maximum detections (maxDet) with the highest scores. And from there get the precision-recall curve for maxDets = 1, 10, 100. Why is this a good metric since it is clearly not the same as the above method (it potentially excludes datapoints)?

在我的示例中，每个图像有大约 3000 个对象.使用 cocoapi 评估结果给出了可怕的回忆，因为它将检测到的对象数量限制为 100.

In my example, I have ~ 3000 objects per image. Evaluating the result using the cocoapi gives terrible recall because it limits the number of detected objects to 100.

出于测试目的，我将评估数据集作为基本事实和检测到的对象(带有一些人工分数)提供.我希望准确率和召回率都很好，这实际上正在发生.但是一旦我输入了 100 多个对象，精度和召回率就会随着检测到的对象"数量的增加而下降.即使他们都是正确的"！这有什么意义?

For testing purposes, I feed the evaluation dataset as the ground truth and the detected objects (with some artificial scores). I would expect precision and recall pretty good, which is actually happening. But as soon as I feed in more than 100 objects, precision and recall go down with increasing number of "detected objects". Even though they are all "correct"! How does that make sense?

推荐答案

我得出的结论是，这正是 cocoapi 定义其度量的方式.这在他们的上下文中可能是有道理的，但我也可以根据我在上面阅读和链接的文章来定义我自己的(这就是我所做的).

I came to the conclusion, that this is just the way that the cocoapi defines its metric. It probably makes sense in their context, but I can as well define my own (which is what I did), based on the articles I read and linked above.

这篇关于理解COCO评估“最大检测"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！