OpenCV图像匹配 - 表单照片与表单模板 [英] OpenCV image matching - form photo vs form template

问题描述

我正在尝试检测照片是否代表填充了数据的预定义公式模板。

我是图像处理和OpenCV的新手，但我的第一次尝试是使用FlannBasedMatcher并比较检测到的关键点数。

有更好的方法吗？

在我的代码中，这是在 getQuadrilateral（） 。

第3步：同形和战争ping

• 使用 findHomography 找到两个表单'bounding rect之间的转换


• 使用 warpPerspective （和单应性<$ c $）扭曲照片的二进制 Mat c> Mat 先前计算过。）

第4步：模板和照片之间的比较

• 扩展模板表格的二进制 Mat 。


• 减去扭曲的二进制 Mat 和扩张的模板表格的二进制 Mat 。

帮助我找回了最大的轮廓。

I'm trying to detect wether a photo represents a predefined formular template filled with data.

I'm new to image processing and OpenCV but my first attempt is to use FlannBasedMatcher and compare the count of keypoints detected.

Is there a better way to do this?

filled-form.jpg

form-template.jpg

import numpy as np
import cv2
from matplotlib import pyplot as plt
MIN_MATCH_COUNT = 10
img1 = cv2.imread('filled-form.jpg',0)          # queryImage
img2 = cv2.imread('template-form.jpg',0) # trainImage
# Initiate SIFT detector
sift = cv2.xfeatures2d.SIFT_create()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)
# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
  if m.distance < 0.7*n.distance:
    good.append(m)
if len(good)>MIN_MATCH_COUNT:
  print "ALL GOOD!" 
else:
  print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
  matchesMask = None

解决方案

I think that using SIFT and a keypoints matcher is the most robust approach to this problem. It should work fine with many different form templates. However, SIFT algorithm being patented, here is another approach that should work well too:

Step 1: Binarize

• Threshold your photo and the template form using THRESH_OTSU tag.
• Invert the two binary result Mats with the bitwise_notfunction.

Step 2: Find the forms' bounding rect

For the two binary Mats from Step 1:

• Find the largest contour.
• Use approxPolyDPto approximate the found contour to a quadrilateral (see picture above).

In my code, this is done inside getQuadrilateral().

Step 3: Homography and Warping

• Find the transformation between the two forms' bounding rect with findHomography
• Warp the photo's binary Mat using warpPerspective (and the homography Mat computed previously).

Step 4: Comparison between template and photo

• Dilate the template form's binary Mat.
• Subtract the warped binary Mat and the dilated template form's binary Mat.

This allows to extract the filled informations. But you can also do it the other way around:

Template form - Dilated Warped Mat

In this case, the result of the subtraction should be totally black. I would then use mean to get the average pixel's value. Finally, if that value is smaller than (let's say) 2, I would assume the form on the photo is matching the template form.

---

Here is the C++ code, it shouldn't be too hard to translate to Python :)

vector<Point> getQuadrilateral(Mat & grayscale)
{
    vector<vector<Point>> contours;
    findContours(grayscale, contours, RETR_EXTERNAL, CHAIN_APPROX_NONE);

    vector<int> indices(contours.size());
    iota(indices.begin(), indices.end(), 0);

    sort(indices.begin(), indices.end(), [&contours](int lhs, int rhs) {
        return contours[lhs].size() > contours[rhs].size();
    });

    vector<vector<Point>> polygon(1);
    approxPolyDP(contours[indices[0]], polygon[0], 5, true);
    if (polygon[0].size() == 4) // we have found a quadrilateral
    {
        return(polygon[0]);
    }
    return(vector<Point>());
}

int main(int argc, char** argv)
{
    Mat templateImg, sampleImg;
    templateImg = imread("template-form.jpg", 0);
    sampleImg = imread("sample-form.jpg", 0);
    Mat templateThresh, sampleTresh;
    threshold(templateImg, templateThresh, 0, 255, THRESH_OTSU);
    threshold(sampleImg, sampleTresh, 0, 255, THRESH_OTSU);

    bitwise_not(templateThresh, templateThresh);
    bitwise_not(sampleTresh, sampleTresh);

    vector<Point> corners_template = getQuadrilateral(templateThresh);
    vector<Point> corners_sample = getQuadrilateral(sampleTresh);

    Mat homography = findHomography(corners_sample, corners_template);

    Mat warpSample;
    warpPerspective(sampleTresh, warpSample, homography, Size(templateThresh.cols, templateThresh.rows));

    Mat element_dilate = getStructuringElement(MORPH_ELLIPSE, Size(8, 8));
    dilate(templateThresh, templateThresh, element_dilate);

    Mat diff = warpSample - templateThresh;

    imshow("diff", diff);

    waitKey(0);

    return 0;
}

I Hope it is clear enough! ;)

P.S. This great answer helped me to retrieve the largest contour.

这篇关于OpenCV图像匹配 - 表单照片与表单模板的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！