使用OpenCV.js检测拐角处带有正方形的框架 [英] Detect a frame with squares in corners with OpenCV.js
问题描述
我一直在使用Javascript和OpenCV.js创建填充表单扫描程序.我基本上想要做的是拍摄一张纸，上面有一张填好的表格的照片，并且能够扫描照片并分析表格中的答案.第一步是实际在图片中找到表单，并应用透视变换获得论文的自顶向下"视图.我要做的是设法获取脚本来检测纸张并应用转换以对其进行良好的扫描.我先应用灰度，然后进行Canny边缘检测，然后对找到的边缘进行迭代，找到最大的4个角的边缘，并假定这一定是我的论文.
I have been playing with creating a filled form scanner with Javascript and OpenCV.js. What I basically want to do is to take a photo of a piece of paper with a filled form on it and be able to scan the photo and analyze the answers in the form. The first step is to actually find the form in the picture and apply perspective transform to get the "top-down view" of the paper. What I have done is I managed to get the script to detect the piece of paper and apply the transform to get it nicely scanned. I did it by applying grayscale, then Canny edge detection, iterated over the found edges and found the largest one with 4 corners and assumed this must be my paper.
这相对来说效果很好，但是脚本时常会混淆纸张的实际内容-有时会检测到其他矩形并将其假定为纸张，有时将纸张作为背景进行拍摄光线和边缘不清晰(对比度不足).后来，当脚本认为已找到论文时，这确实破坏了我的流程，但实际上是另外一回事.我希望在纸张检测部分进行改进，以便始终可以确定检测到正确的内容.我想-让我们在表单周围添加一个自定义框架，这样会更容易检测并在角落添加一些正方形(以再次检查找到的框架是否是我要查找的框架的100％).
This works relatively well, but every now and then the script gets confused as to what the paper actually is - sometimes there are other rectangles that are detected and assumed to be paper, sometimes the background on which the paper is photographed is very light and edges aren't clear (not enough contrast). That really destroys my flow later when the script thinks it has found the paper, but really it's something else. I would like to improve on this paper detection part so I can be always sure that the right thing has been detected. I thought - let's add a custom frame around the form, which will be easier to detect and add some squares in the corners (to double check if the found frame is 100% the one I'm looking for).
所以我创建了这样的东西:
So I have created something like this:
现在，我希望能够检测框架的角并确保角落中有实心"正方形，以确保这是我要查找的框架的100％.您能提供关于如何使用openCV实现它的建议吗?这是正确的方法吗?谢谢！
Now I would like to be able to detect the corners of the frame and make sure the "filled" squares are there in the corners to be sure that this is 100% the frame I am looking for. Can you please advice on how to achieve it with openCV? Is this the right way to go? Thanks!
推荐答案
我之前也曾研究过类似的问题.我使用OpenCV的C ++实现，但是我为您提供了一些技巧.
I've worked on a similar problem before. I work with the C++ implementation of OpenCV, but I have some tips for you.
细分论文
To achieve a better segmentation, consider trying Image Quantization. This technique segments the image in N clusters, that is, it groups pixels of similar colors into a group. This group is then represented by one color.
此技术相对于其他纯 binary thresholding 的优势在于，它可以识别多个颜色分布，这些颜色分布将被归为 N个簇.签出(抱歉，链接，我还不能发布直接图像):
The advantage of this technique over other, say, pure binary thresholding, is that it can identify multiple color distributions – those that will be grouped in N clusters. Check it out (Sorry for the links, I'm not allowed -yet- to post direct images):
这将帮助您更好地分割论文.该实现使用称为"K-means" 的聚类算法(稍后会详细介绍).在我的示例中，我尝试了3个群集和5种算法运行"(或尝试，因为K-means通常运行不止一次).
This will help you get a better segmentation of your paper. The implementation uses the clustering algorithm known as "K-means" (more of this later). In my example, I tried 3 clusters and 5 algorithms "runs" (or attempts, as K-means is often run more than one time).
cv::Mat imageQuantization( cv::Mat inputImage, int numberOfClusters = 3, int iterations = 5 ){
//step 1 : map the src to the samples
cv::Mat samples(inputImage.total(), 3, CV_32F);
auto samples_ptr = samples.ptr<float>(0);
for( int row = 0; row != inputImage.rows; ++row){
auto src_begin = inputImage.ptr<uchar>(row);
auto src_end = src_begin + inputImage.cols * inputImage.channels();
//auto samples_ptr = samples.ptr<float>(row * src.cols);
while(src_begin != src_end){
samples_ptr[0] = src_begin[0];
samples_ptr[1] = src_begin[1];
samples_ptr[2] = src_begin[2];
samples_ptr += 3; src_begin +=3;
}
}
//step 2 : apply kmeans to find labels and centers
int clusterCount = numberOfClusters; //Number of clusters to split the set by
cv::Mat labels;
int attempts = iterations; //Number of times the algorithm is executed using different initial labels
cv::Mat centers;
int flags = cv::KMEANS_PP_CENTERS;
cv::TermCriteria criteria = cv::TermCriteria( CV_TERMCRIT_ITER | CV_TERMCRIT_EPS,
10, 0.01 );
//the call to kmeans:
cv::kmeans( samples, clusterCount, labels, criteria, attempts, flags, centers );
//step 3 : map the centers to the output
cv::Mat clusteredImage( inputImage.size(), inputImage.type() );
for( int row = 0; row != inputImage.rows; ++row ){
auto clusteredImageBegin = clusteredImage.ptr<uchar>(row);
auto clusteredImageEnd = clusteredImageBegin + clusteredImage.cols * 3;
auto labels_ptr = labels.ptr<int>(row * inputImage.cols);
while( clusteredImageBegin != clusteredImageEnd ){
int const cluster_idx = *labels_ptr;
auto centers_ptr = centers.ptr<float>(cluster_idx);
clusteredImageBegin[0] = centers_ptr[0];
clusteredImageBegin[1] = centers_ptr[1];
clusteredImageBegin[2] = centers_ptr[2];
clusteredImageBegin += 3; ++labels_ptr;
}
}
//return the output:
return clusteredImage;
}
请注意，该算法还会产生两个附加矩阵.标签" 是用标识其簇的整数标记的实际像素.中心" 是每个群集的平均值值.
Note that the algorithm also produces two additional matrices. "Labels" are the actual pixels labeled with an integer that identifies their cluster. "Centers" are the mean values of each cluster.
检测边缘
现在，在此分割的图像上运行 Edge Detector 很简单.让我们尝试一下Canny.当然，您可以调整参数.在这里，我尝试了 Lower Threshold 0f 30和 Upper Threshold 为90.相当标准，只需确保Upper Threshold遵循= 3 * LowerThreshold的条件即可，例如每个Canny的建议.结果是:
Now, it is trivial to run an Edge Detector on this segmented image. Let’s try Canny. The parameters, of course, can be adjusted by you. Here, I tried a Lower Threshold 0f 30, and an Upper Threshold of 90. Pretty standard, just make sure the Upper Threshold follows the condition that = 3 * LowerThreshold, as per Canny suggestions. This is the result:
cv::Mat testEdges;
float lowerThreshold = 30;
float upperThreshold = 3 * lowerThreshold;
cv::Canny( testSegmented, testEdges, lowerThreshold, upperThreshold );
检测行
好.是否要检测边缘检测器产生的线条?在这里，至少有2个选项.第一个也是最简单的方法:使用霍夫行检测器.但是，正如您肯定已经看到的那样，调整Hough来查找实际要查找的行可能很困难.
Nice. Want to detect the lines produced by the edge detector? Here, there are at least 2 options. The first and most straightforward: Use Hough’s Line Detector. However, as you surely have seen, tuning Hough to find the lines you are actually looking for could be difficult.
过滤Hough返回的线的一种可能的解决方案是运行角度过滤器" ，因为我们只在寻找(接近)垂直和水平线.您还可以按长度过滤线.
One possible solution to filter the lines returned by Hough is to run an "angle filter", as we are looking for only (close to) vertical and horizontal lines. You can also filter the lines by length.
此代码段给出了这个想法，您需要实际实现过滤器://运行霍夫的线检测器:cv :: HoughLinesP(grad，linesP，1，CV_PI/180，minVotes，minLineLength，maxLineGap);
This code snippet gives out the idea, you need to actually implement the filter: // Run Hough's Line Detector: cv::HoughLinesP(grad, linesP, 1, CV_PI/180, minVotes, minLineLength, maxLineGap );
// Process the points (lines)
for( size_t i = 0; i < linesP.size(); i++ ) //points are stored in linesP
{
//get the line
cv::Vec4i l = linesP[i]; //get the line
//get the points:
cv::Point startPoint = cv::Point( l[0], l[1] );
cv::Point endPoint = cv::Point( l[2], l[3] );
//filter horizontal & vertical:
float dx = abs(startPoint.x - endPoint.x);
float dy = abs(startPoint.y - endPoint.y);
//angle filtering, delta y and delta x
if ( (dy < maxDy) || (dx < maxDx) ){
//got my target lines!
}
}
在上面的代码中，我实际上是在处理线组件，而不是角度.因此，我的角度"限制由2个最小组件长度定义: maxDy -最大增量"y轴的长度，以及x轴的 maxDx .
In the code above, I'm actually working with line components, instead of angles. So, my "angle" restrictions are defined by 2 minimum component lengths: maxDy - the maximum "delta" length in the y axis, as well as maxDx for the x axis.
另一种用于线条检测的解决方案是利用您仅查看具有角落或它们之间约90度的线条这一事实.您可以运行形态过滤器，通过 hit or miss 操作检测这些模式":)
The other solution for line detection is to exploit the fact you are only looking lines that have CORNERS or about 90 degrees between them. You can run a morphological filter to detect these "patterns" via a hit or miss operation :)
无论如何，回到霍夫，这是我在没有进行太多参数调整的情况下并且在应用了角度/长度线滤波器之后得到的检测结果:
Anyway, back to Hough, this is the detection I get without too much parameter tuning and after applying the angle/length line filter:
很酷.绿点代表线的起点和终点.如您所见，其中有很多.我们如何组合"它们?如果我们计算这些点的平均值怎么办?好的，但是我们应该获得PER象限"行的均值.如下图所示，我将输入图像分为4个象限(黄线):
Cool. The green dots represent the start and endpoints of the lines. As you see, there’s a bunch of them. How can we "combine" them? What if we compute the mean of those points? Ok, but we should get the mean of the lines PER "quadrant". As you see in the following figure, I’ve divided the input image into 4 quadrants (the yellow lines):
每个象限-希望-将包含描述纸角的点.对于这些象限中的每个象限，检查哪些点落在给定象限上并计算它们的平均值.那是一般的想法.
Each of the quadrants -hopefully- will contain the points that describe the corner of the paper. For each of these quadrants, check which points fall on a given quadrant and compute the mean of them. That’s the general idea.
要编写的代码很多.幸运的是，如果我们稍微研究一下问题，我们可以看到所有绿色的点在某些非常明确的区域(或我们前面所说的象限")中趋向于聚集.输入K-再次表示.
That’s quite some code to write. Fortunately, if we study the problem for a bit, we can see that all the green dots tend to CLUSTER in some very defined regions (or, as we said earlier, the "quadrants".) Enter K-means again.
无论如何，K均值将对相似值的数据进行分组.它可以是像素，可以是空间点，也可以是任意点，只需为其提供数据集和所需的聚类数，它就会吐出找到的聚类和有效的聚类手段-很好！
K-means will group data of similar value no matter what. It can be pixels, it can be spatial points, it can be whatever, just give it the data set and the number of clusters you want, and it will spit out the clusters found and THE MEANS OF SAID CLUSTERS – NICE!
如果我用Hough返回的线点运行K-means，我得到的结果将显示在上一张图像中.我也舍弃了离均值太远的点.点的平均值是通过点的中心"返回的.矩阵，在这里它们以橙色呈现-相当接近！
If I run K-means with the line points returned by Hough I get the result shown in the last image. I've also discarded points that are too far from the mean. The means of the points are returned via the "centers" matrix and here they are rendered in orange- that’s quite close!
希望其中一些对您有所帮助！:)
Hope that some of this helps you! :)
这篇关于使用OpenCV.js检测拐角处带有正方形的框架的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
