检测图像中的乐高底板 [英] Detecting a lego baseplate in an image

问题描述

我希望我的代码在图像中找到一个正方形的乐高板的角，如所附的图像.

I want my code to find the corners of a square lego plate in an image like the one attached.

我还想找到它的尺寸，即两个尺寸中斑点"的数量(在所附图片中为48x48).

I also want to find its dimensions, i.e. the number of "blops" in both dimensions (48x48 in the attached image).

我目前正在寻找检测单个斑点"的方法，到目前为止，结果是相当不错的:将模糊，自适应阈值，findContours和基于区域的选择结合起来，可以在第二幅附加图像中找到轮廓(着色是随机的) ).

I am currently looking at detecting the individual "blops", and the result so far is pretty good: a combination of blur, adaptiveThreshold, findContours and selection based on area finds the contours rendered in the second attached image (coloring is random).

我现在正在寻找一种算法来查找这些轮廓(或它们的中点)所代表的网格"，但我缺少google fu.有任何想法吗?

I'm now looking for an algorithm to find the "grid" losely represented by these contours (or their mid-points), but I lack the google fu. Any ideas?

(也欢迎提出不同方法的建议.)

(Suggestions for different approaches are also very welcome.)

(示例图像显示了放在角落的砖块，如果有帮助，算法可以预期到这一点.)

(The sample image shows bricks placed in the corners - an algorithm could expect this, if it helps.)

(示例图像的背景非常狂野.如果可能的话，我更愿意解决这个问题.)

(The sample image has a rather wild background. I'd prefer to cope with that, if possible.)

2016年7月8日更新:我正在尝试编写一种算法，用于寻找形成线的相邻轮廓的条纹.该算法应该能够找到许多这样的算法，并从中推论整个板的形式，即使是有角度的情况也是如此.如果有效，将进行更新...

Update 8 July 2016: I'm trying to write an algorithm that looks for streaks of adjacent contours forming lines. The algo should be able to find a number of these and, from that, deduce the form of the whole plate, even with perspective. Will update if it works...

2017年12月更新:尽管上述算法有点不可预测，但上述算法仍然有效.另外，我在透视(添加厚"乐高积木改变了表面)和颜色识别(阴影，相机特性等)方面也遇到了问题.这项工作暂时搁置.如果我恢复它，我将尝试将固定的相机位置固定在平板上方，并保持一致的灯光.

Update December 2017: The above algorithm sort of worked, although it was a bit too unpredictable. Also I got problems with perspective (adding a "thick" lego brick changes the surface) and color recognition (shadows, camera peculiarities etc). This endeavor is on hold for now. If I resume it I will try with fixed camera positions immediately above the plate and consistent lights.

推荐答案

这是一种使用颜色阈值的潜在方法.想法是在假设底板为灰色的情况下，使用下限和上限将图像转换为HSV格式，然后将其转换为颜色阈值.这将为我们提供蒙版图像.从这里开始，我们进行变形以消除噪声，找到轮廓并为最大轮廓排序.接下来，我们获得旋转的边界框坐标，并将其绘制到新的空白蒙版上.最后，我们按位运算-然后使用带有输入图像的蒙版获得结果.要找到拐角坐标，我们可以使用cv2.goodFeaturesToTrack()在遮罩上找到点.看看 和

Here's a potential approach using color thresholding. The idea is to convert the image to HSV format then color threshold using a lower and upper bound with the assumption that the baseplate is in gray. This will give us a mask image. From here we morph open to remove noise, find contours, and sort for the largest contour. Next we obtain the rotated bounding box coordinates and draw this onto a new blank mask. Finally we bitwise-and the mask with the input image to get our result. To find the corner coordinates, we can use cv2.goodFeaturesToTrack() to find the points on the mask. Take a look at how to find accurate corner positions of a distorted rectangle from blurry image in python? and Shi-Tomasi Corner Detector & Good Features to Track for more details

以下是每个步骤的可视化结果:

Here's a visualization of each step:

我们加载图像，转换为HSV格式，定义上下限，并使用cv2.inRange()

We load the image, convert to HSV format, define a lower/upper bound, and perform color thresholding using cv2.inRange()

import cv2
import numpy as np

# Load image, convert to HSV, and color threshold
image = cv2.imread('1.png')
blank_mask = np.zeros(image.shape, dtype=np.uint8)
original = image.copy()
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 109])
upper = np.array([179, 36, 255])
mask = cv2.inRange(hsv, lower, upper)

接下来，我们使用cv2.getStructuringElement()创建一个矩形内核，并使用cv2.morphologyEx()执行形态运算.此步骤将去除小的噪音颗粒.

Next we create a rectangular kernel using cv2.getStructuringElement() and perform morphological operations using cv2.morphologyEx(). This step removes small particles of noise.

# Morph open to remove noise
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel, iterations=1)

在这里，我们使用cv2.findContours()在遮罩上找到轮廓，并使用轮廓区域进行滤波以获得最大轮廓.然后，我们使用cv2.minAreaRect()和cv2.boxPoints()获得旋转的绑定框坐标，然后使用cv2.fillPoly()将其绘制到新的空白蒙版上.此步骤为我们提供了底板的完美"外轮廓.这是检测到的外部轮廓，以绿色突出显示，并显示了蒙版.

From here we find contours on the mask using cv2.findContours() and filter using contour area to obtain the largest contour. We then obtain the rotated boding box coordinates using cv2.minAreaRect() and cv2.boxPoints() then draw this onto a new blank mask with cv2.fillPoly(). This step gives us a "perfect" outer contour of the baseplate. Here's the detected outer contour highlighted in green and the resulting mask.

# Find contours and sort for largest contour
cnts = cv2.findContours(opening, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[0]

# Obtain rotated bounding box and draw onto a blank mask
rect = cv2.minAreaRect(cnts)
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(image,[box],0,(36,255,12),3)
cv2.fillPoly(blank_mask, [box], (255,255,255))

最后，我们按位运算-然后使用原始输入图像的蒙版获得结果.根据需要，可以将背景更改为黑色或白色.

Finally we bitwise-and the mask with our original input image to obtain our result. Depending on what you need, you can change the background to black or white.

# Bitwise-and mask with input image 
blank_mask = cv2.cvtColor(blank_mask, cv2.COLOR_BGR2GRAY)
result = cv2.bitwise_and(original, original, mask=blank_mask)
# result[blank_mask==0] = (255,255,255) # Color background white

要检测拐角坐标，可以使用cv2.goodFeaturesToTrack().这是检测到的角落，以紫色突出显示:

To detect the corner coordinates, we can use cv2.goodFeaturesToTrack(). Here's the detected corners highlighted in purple:

坐标:

(91.0, 525.0)
(463.0, 497.0)
(64.0, 152.0)
(436.0, 125.0)

# Detect corners
corners = cv2.goodFeaturesToTrack(blank_mask, maxCorners=4, qualityLevel=0.5, minDistance=150)
for corner in corners:
    x,y = corner.ravel()
    cv2.circle(image,(x,y),8,(155,20,255),-1)
    print("({}, {})".format(x,y))

完整代码

import cv2
import numpy as np

# Load image, convert to HSV, and color threshold
image = cv2.imread('1.png')
blank_mask = np.zeros(image.shape, dtype=np.uint8)
original = image.copy()
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 109])
upper = np.array([179, 36, 255])
mask = cv2.inRange(hsv, lower, upper)

# Morph open to remove noise
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel, iterations=1)

# Find contours and sort for largest contour
cnts = cv2.findContours(opening, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[0]

# Obtain rotated bounding box and draw onto a blank mask
rect = cv2.minAreaRect(cnts)
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(image,[box],0,(36,255,12),3)
cv2.fillPoly(blank_mask, [box], (255,255,255))

# Bitwise-and mask with input image 
blank_mask = cv2.cvtColor(blank_mask, cv2.COLOR_BGR2GRAY)
result = cv2.bitwise_and(original, original, mask=blank_mask)
result[blank_mask==0] = (255,255,255) # Color background white

# Detect corners
corners = cv2.goodFeaturesToTrack(blank_mask, maxCorners=4, qualityLevel=0.5, minDistance=150)
for corner in corners:
    x,y = corner.ravel()
    cv2.circle(image,(x,y),8,(155,20,255),-1)
    print("({}, {})".format(x,y))

cv2.imwrite('mask.png', mask)
cv2.imwrite('opening.png', opening)
cv2.imwrite('blank_mask.png', blank_mask)
cv2.imwrite('image.png', image)
cv2.imwrite('result.png', result)
cv2.waitKey()

这篇关于检测图像中的乐高底板的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！