计算物体检测混淆矩阵的正确方法是什么? [英] What's the correct way to compute a confusion matrix for object detection?

问题描述

我正在尝试为我的对象检测模型计算混淆矩阵.然而，我似乎偶然发现了一些陷阱.我目前的方法是将每个预测框与每个地面实况框进行比较.如果他们的 IoU > 某个阈值，我会将预测插入到混淆矩阵中.插入后，我删除预测列表中的元素并移至下一个元素.

因为我也希望将错误分类的建议插入混淆矩阵中，所以我将 IoU 低于阈值的元素视为与背景混淆.我目前的实现是这样的:

def insert_into_conf_m(true_labels、predicted_labels、true_boxes、predicted_boxes):匹配_gts = []对于我在范围内(len(true_labels)):j = 0而 len(predicted_labels) != 0:如果 j >= len(predicted_boxes):休息如果 bb_intersection_over_union(true_boxes[i],predicted_boxes[j]) >= 0.7:conf_m[true_labels[i]][predicted_labels[j]] += 1del Predicted_boxes[j]德尔预测标签[j]别的:j += 1match_gts.append(true_labels[i])如果 len(predicted_labels) == 0:休息# 如果有任何建议都不匹配的groundtruth框# 它们被视为模型将它们归类为背景如果 len(true_labels) >len(matched_gts):true_labels = [i for i in true_labels if not i inmatched_gts 或matched_gts.remove(i)]对于我在范围内(len(true_labels)):conf_m[true_labels[i]][0] += 1# 处理所有与任何真实框都没有 IoU 的检测# 就好像这个区域的真实标签是背景 (0)如果 len(predicted_labels) != 0:对于范围内的 j(len(predicted_labels)):conf_m[0][predicted_labels[j]] += 1

行归一化矩阵如下所示:

[0.0, 0.36, 0.34, 0.30][0.0, 0.29, 0.30, 0.41][0.0, 0.20, 0.47, 0.33][0.0, 0.23, 0.19, 0.58]

有没有更好的方法来为物体检测系统生成混淆矩阵?或者其他更合适的指标?

解决方案

这里有一个脚本来计算TensorFlow Object Detection API 生成的 detections.record 文件中的混淆矩阵.这是文章 解释这个脚本的工作原理.

总而言之，这里是文章中的算法大纲:

<块引用>

1. 对于每个检测记录，算法从输入文件中提取真实值框和类，以及检测到的盒子、班级和分数.

2. 仅考虑得分大于或等于 0.5 的检测.任何低于此值的都将被丢弃.

3. 对于每个ground-truth box，算法会为每个检测到的box生成IoU(交集).如果找到匹配项两个盒子的 IoU 都大于或等于 0.5.

4. 对匹配列表进行修剪以删除重复项(与多个检测框匹配的真实框，反之亦然).如果存在重复项，始终选择最佳匹配(更大的 IoU).

5. 更新混淆矩阵以反映真实情况和检测结果之间的匹配结果.

6. 属于地面实况但未被检测到的对象计入矩阵的最后一列(在对应于真实类).检测到但不属于的对象混淆矩阵计入矩阵的最后一行(在与检测到的类别对应的列).

您还可以查看脚本了解更多信息.

I am trying to compute a confusion matrix for my object detection model. However I seem to stumble across some pitfalls. My current approach is to compare each predicted box with each groundtruth box. If they have a IoU > some threshold, I insert the predictions into the confusion matrix. After the insertion I delete the element in the predictions list and move on to the next element.

Because I also want the misclassified proposals to be inserted in the confusion matrix, I treat the elements with IoU lower than the threshold as confusion with background. My current implemention looks like this:

def insert_into_conf_m(true_labels, predicted_labels, true_boxes, predicted_boxes):
    matched_gts = []
    for i in range(len(true_labels)):
        j = 0
        while len(predicted_labels) != 0:
            if j >= len(predicted_boxes):
                break
            if bb_intersection_over_union(true_boxes[i], predicted_boxes[j]) >= 0.7:
                conf_m[true_labels[i]][predicted_labels[j]] += 1
                del predicted_boxes[j]
                del predicted_labels[j]
            else:
                j += 1
        matched_gts.append(true_labels[i])
        if len(predicted_labels) == 0:
            break
    # if there are groundtruth boxes that are not matched by any proposal
    # they are treated as if the model classified them as background
    if len(true_labels) > len(matched_gts):
        true_labels = [i for i in true_labels if not i in matched_gts or matched_gts.remove(i)]
        for i in range(len(true_labels)):
            conf_m[true_labels[i]][0] += 1

    # all detections that have no IoU with any groundtruth box are treated
    # as if the groundtruth label for this region was Background (0)
    if len(predicted_labels) != 0:
        for j in range(len(predicted_labels)):
            conf_m[0][predicted_labels[j]] += 1

The row-normalized matrix looks like this:

[0.0, 0.36, 0.34, 0.30]
[0.0, 0.29, 0.30, 0.41]
[0.0, 0.20, 0.47, 0.33]
[0.0, 0.23, 0.19, 0.58]

Is there a better way to generate the confusion matrix for an object detection system? Or any other metric that is more suitable?

解决方案

Here is a script to compute the confusion matrix from the detections.record file generated by the TensorFlow Object Detection API. Here is the article explaining how this script works.

In summary, here is the outline of the algorithm from the article:

1. For each detection record, the algorithm extracts from the input file the ground-truth boxes and classes, along with the detected boxes, classes, and scores.

2. Only detections with a score greater or equal than 0.5 are considered. Anything that’s under this value is discarded.

3. For each ground-truth box, the algorithm generates the IoU (Intersection over Union) with every detected box. A match is found if both boxes have an IoU greater or equal than 0.5.

4. The list of matches is pruned to remove duplicates (ground-truth boxes that match with more than one detection box or vice versa). If there are duplicates, the best match (greater IoU) is always selected.

5. The confusion matrix is updated to reflect the resulting matches between ground-truth and detections.

6. Objects that are part of the ground-truth but weren’t detected are counted in the last column of the matrix (in the row corresponding to the ground-truth class). Objects that were detected but aren’t part of the confusion matrix are counted in the last row of the matrix (in the column corresponding to the detected class).

You can also take a look at the script for more information.

这篇关于计算物体检测混淆矩阵的正确方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！