图像处理:实时FedEx徽标检测器的算法改进 [英] Image Processing: Algorithm Improvement for Real-Time FedEx Logo Detector

问题描述

我一直在从事一个涉及用于徽标检测的图像处理的项目.具体来说，目标是为实时FedEx卡车/徽标检测器开发一个自动化系统，该系统可从IP摄像机流中读取帧并发送检测通知.这是运行中的系统的示例，其中已识别的徽标被包围在绿色矩形中.

I've been working on a project involving image processing for logo detection. Specifically, the goal is to develop an automated system for a real-time FedEx truck/logo detector that reads frames from a IP camera stream and sends a notification on detection. Here's a sample of the system in action with the recognized logo surrounded in the green rectangle.

对项目的一些限制:

• 使用原始OpenCV(不使用深度学习，人工智能或训练有素的神经网络)
• 图像背景可能很吵
• 图像的亮度变化很大(早晨，下午，晚上)
• FedEx卡车/徽标可以具有任何 scale ， rotation 或方向，因为它可以停在人行道上的任何地方
• 根据一天中的不同时间，徽标可能会变得模糊或阴影深浅
• 同一框架中可能还有许多其他具有相似尺寸或颜色的车辆
• 实时检测(来自IP摄像机的〜25 FPS)
• IP摄像机固定在一个位置，联邦快递卡车将始终保持相同的方向(绝不向后或上下颠倒)
• 联邦快递卡车将始终是红色"版本，而不是

• Uses raw OpenCV (no deep learning, AI, or trained neural networks)
• Image background can be noisy
• The brightness of the image can vary greatly (morning, afternoon, night)
• The FedEx truck/logo can have any scale, rotation, or orientation since it could be parked anywhere on the sidewalk
• The logo could potentially be fuzzy or blurry with different shades depending on the time of day
• There may be many other vehicles with similar sizes or colors in the same frame
• Real-time detection (~25 FPS from IP camera)
• The IP camera is in a fixed position and the FedEx truck will always be in the same orientation (never backwards or upside down)
• The Fedex Truck will always be the "red" variation instead of the "green" variation

当前实施/算法

我有两个线程:

• 线程#1-使用cv2.VideoCapture()从IP摄像机捕获帧并调整帧大小以进行进一步处理.由于cv2.VideoCapture()正在阻塞，因此决定通过减少I/O延迟来在单独的线程中处理抓取帧以提高FPS.通过专门用于捕获帧的独立线程，这将允许主处理线程始终具有可用于执行检测的帧.
• 线程#2-使用颜色阈值和轮廓检测来检测FedEx徽标的主要处理/检测线程.

• Thread #1 - Captures frames from the IP camera using cv2.VideoCapture() and resizes frame for further processing. Decided to handle grabbing frames in a separate thread to improve FPS by reducing I/O latency since cv2.VideoCapture() is blocking. By dedicating an independent thread just for capturing frames, this would allow the main processing thread to always have a frame available to perform detection on.
• Thread #2 - Main processing/detection thread to detect FedEx logo using color thresholding and contour detection.

总体伪算法

For each frame:
    Find bounding box for purple color of logo
    Find bounding box for red/orange color of logo
    If both bounding boxes are valid/adjacent and contours pass checks:
        Combine bounding boxes
        Draw combined bounding boxes on original frame
        Play sound notification for detected logo

用于徽标检测的颜色阈值

对于颜色阈值，我为紫色和红色定义了HSV(低，高)阈值以检测徽标.

For color thresholding, I have defined HSV (low, high) thresholds for purple and red to detect the logo.

colors = {
    'purple': ([120,45,45], [150,255,255]),
    'red': ([0,130,0], [15,255,255]) 
}

要找到每种颜色的边界框坐标，请遵循以下算法:

To find the bounding box coordinates for each color, I follow this algorithm:

• 模糊框架
• 用内核侵蚀并扩大框架，以消除背景噪声
• 将帧从BGR转换为HSV颜色格式
• 使用具有设置的颜色阈值的HSV上下边界在帧上执行蒙版
• 在蒙版中找到最大轮廓并获取边界坐标

执行遮罩后，我获得了徽标的这些孤立的紫色(​​左)和红色(右)部分.

After performing a mask, I obtain these isolated purple (left) and red (right) sections of the logo.

假阳性检查

现在我有了两个遮罩，我将执行检查以确保找到的边界框实际上形成徽标.为此，我使用 cv2.matchShapes() 比较两个轮廓并返回显示相似度的度量.结果越低，匹配度越高.另外，我使用cv2.pointPolygonTest()来查找图像中的点与轮廓之间的最短距离，以进行其他验证.我的误报过程涉及:

Now that I have the two masks, I perform checks to ensure that the found bounding boxes actually form a logo. To do this, I use cv2.matchShapes() which compares the two contours and returns a metric showing the similarity. The lower the result, the higher the match. In addition, I use cv2.pointPolygonTest() which finds the shortest distance between a point in the image and a contour for additional verification. My false positive process involves:

• 检查边界框是否有效
• 基于两个边界框的相对接近度确保它们相邻

如果边界框通过了邻接和相似性度量测试，则边界框将被合并并触发FedEx通知.

If the bounding boxes pass the adjacency and similarity metric test, the bounding boxes are combined and a FedEx notification is triggered.

结果

此检查算法并不十分可靠，因为存在许多误报和检测失败的情况.例如，这些误报被触发.

This check algorithm is not really robust as there are many false positives and failed detections. For instance, these false positives were triggered.

虽然这种颜色阈值和轮廓检测方法在徽标清晰的基本情况下有效，但在某些领域却严重缺乏:

While this color thresholding and contour detection approach worked in basic cases where the logo was clear, it was severely lacking in some areas:

• 必须在每个帧上计算边界框会带来延迟问题
• 偶尔会检测到不存在徽标的情况
• 亮度和时间对检测精度有很大的影响
• 当徽标倾斜时，可以进行颜色阈值检测，但是由于检查算法而无法检测到徽标.

任何人都可以帮助我改善算法或提出替代检测策略的建议吗?由于颜色阈值高度依赖于精确校准，还有其他方法可以执行此检测吗?如果可能的话，我不想使用颜色阈值和多层滤镜，因为它不是很可靠.任何见解或建议，我们将不胜感激！

Would anyone be able to help me improve my algorithm or suggest alternative detection strategies? Is there any other way to perform this detection since color thresholding is highly dependent on exact calibration? If possible, I would like to move away from color thresholding and the multiple layers of filters since it's not very robust. Any insight or advice is greatly appreciated!

推荐答案

您可能想看一下功能匹配.目标是在两个图像(模板图像和嘈杂的图像)中找到特征并将其匹配.这样一来，您就可以在嘈杂的图像(相机图像)中找到模板(徽标).

You might want to take a look at feature matching. The goal is to find features in two images, a template image, and a noisy image and match them. This would allow you to find the template (the logo) in the noisy image (the camera image).

从本质上讲，一种功能是人们会在图像中发现的有趣事物，例如角落或开放空间.我建议使用尺度不变特征变换(SIFT)作为特征检测算法.我建议使用SIFT的原因是，它对于图像的平移，缩放和旋转是不变的，对于照明变化部分不变，并且对于局部几何失真也很稳定.这符合您的规范.

A feature is, in essence, things that humans would find interesting in an image, such as corners or open spaces. I would recommend using a scale-invariant feature transform (SIFT) as a feature detection algorithm. The reason I suggest using SIFT is that it is invariant to image translation, scaling, and rotation, partially invariant to illumination changes and robust to local geometric distortion. This matches your specification.

我使用从这样做时，您会注意到很多功能确实落在了FedEx徽标上(上图).

You will notice when doing this that a large number of the features do land on the FedEx logo (Above).

我要做的下一件事是尝试将视频供稿中的功能与FedEx徽标中的功能进行匹配.我使用FLANN功能匹配器完成了此操作.您可能采用了许多方法(包括蛮力)，但是由于您正在处理视频提要，因此这可能是您的最佳选择.以下代码的灵感来自 OpenCV文档有关功能匹配的信息:

The next thing I did was try matching the features from the video feed to the features in the FedEx logo. I did this using the FLANN feature matcher. You could have gone with many approaches (including brute force) but because you are working on a video feed this is probably your best option. The code below is inspired from the OpenCV docs on feature matching:

import numpy as np
import cv2
from matplotlib import pyplot as plt

logo = cv2.imread('logo.jpg', 0) # query Image
img = cv2.imread('main2.jpg',0)  # target Image


# Create the sift object
sift = cv2.xfeatures2d.SIFT_create(700)

# Find keypoints and descriptors directly
kp1, des1 = sift.detectAndCompute(img, None)
kp2, des2 = sift.detectAndCompute(logo,None)

# FLANN parameters
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=50)   # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(des1,des2,k=2)

# Need to draw only good matches, so create a mask
matchesMask = [[0,0] for i in range(len(matches))]

# ratio test as per Lowe's paper
for i,(m,n) in enumerate(matches):
    if m.distance < 0.7*n.distance:
        matchesMask[i]=[1,0]

# Draw lines
draw_params = dict(matchColor = (0,255,0),
                   singlePointColor = (255,0,0),
                   matchesMask = matchesMask,
                   flags = 0)


# Display the matches
img3 = cv2.drawMatchesKnn(img,kp1,logo,kp2,matches,None,**draw_params)
plt.imshow(img3, )
plt.show()

使用此功能，我可以匹配以下功能，如下所示.您会发现有异常值.但是大多数功能都可以匹配:

Using this I was able to get the following features matched as seen below. You will notice that there are outliers. However the majority of features match:

最后的步骤将是简单地在此图像周围绘制一个边界框.我会将您链接到另一个堆栈溢出问题类似，但与球探测器.这是使用 OpenCV文档获取边框的另一种方法.

The final step would then to be to simply draw a bounding box around this image. I will link you to another stack overflow question which does something similar but with the orb detector. Here is another way to get a bounding box using the OpenCV docs.

我希望这会有所帮助！

这篇关于图像处理:实时FedEx徽标检测器的算法改进的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！