使用OpenCV和Matlab从校准的图像和三角形网格生成深度图 [英] Depth map from calibrated image and triangular mesh using OpenCV and Matlab

问题描述

我想使用从Matlab 2014b调用的OpenCV(使用OpenCV绑定)从校准的图像和三角形网格中提取深度图.我是Matlab的普通用户，但对OpenCV还是陌生的.我有以下输入内容:

I want to extract a depth map from a calibrated image and triangular mesh using OpenCV called from Matlab 2014b (using the OpenCV bindings). I am a regular user of Matlab but am new to OpenCV. I have the following inputs:

im-场景的未失真RGB图像

im - an undistorted RGB image of the scene

或-摄像机位置矢量

R-描述相机姿态的旋转矩阵

R - rotation matrix describing camera pose

点-代表场景的nx3个三角形网格顶点列表

points - nx3 triangular mesh vertex list representing the scene

faces-mx3三角形网格面列表

faces - mx3 triangular mesh face list

EFL-图像有效焦距(以像素为单位)

EFL - image effective focal length in pixels

我已经编写了一个本机Matlab光线跟踪引擎来从这些输入中提取深度图，但这非常慢，并且会遭受较大的重投影误差(我想将OpenCV函数的结果与我自己的结果进行比较，以确定是否存在这些误差与我的实现有关，或者仅仅是相机校准不正确).

I have written a native Matlab ray tracing engine to extract a depth map from these inputs, but this is quite slow and suffers from high reprojection errors (I want to compare the results from OpenCV functions to my own to establish if these errors relate to my implementation or just a camera calibration inaccuracies).

如何使用Matlab调用的OpenCV从这些输入中获取深度图?

How can a depth map be obtained from these inputs using OpenCV called from Matlab?

任何帮助将不胜感激

谢谢

托马斯

推荐答案

拟议策略

您可以将网格中的顶点投影到2D像素坐标中(使用经过校准的相机模型).然后，对于每个面，您可以找到由其投影顶点形成的2D三角形中包含的所有像素中心(晶格点).在重叠的情况下，您可能必须跟踪哪个三角形最接近.现在，您知道哪张脸与每个像素相对应.除非网格物体的分辨率比图像的分辨率高得多，否则这应该非常快.

You could project the vertices from your mesh into 2D pixel coordinates (using your calibrated camera model). Then for each face, you can find all of the pixel centres (lattice points) contained in the 2D triangle formed by its projected vertices. You may have to keep track of which triangle is the nearest in the case of overlap. Now you know which face corresponds to each pixel. This should be very fast unless your mesh is much higher resolution than your image.

然后，您可以使用相机模型找到与每个像素相对应的3D射线，并将该射线与该像素的已知面孔相交以计算深度(声音就像您已经做过这一部分一样).既然您已经知道飞机，这也不应该花太长时间.

You can then find the 3D ray corresponding to each pixel using the camera model, and intersect the ray with the known face for that pixel to calculate the depth (sounds like you already did this part). This shouldn't take too long either, now that you know the plane.

有关相机投影的更多信息

OpenCV具有使用相机的良好资源模型(如下). 基本上，您可以将3D点M'投影到像素坐标m'上.这就是将顶点投影到像素位置的方式.从另一个方向看，比例是不可恢复的-您得到的是射线M'/s而不是点M'.您要寻找的深度是s，这是相机框架中3D点的Z坐标.如果网格位于以相机为中心的框架中(X向右，Y向下，Z向外)，R = Identity和t = 0.如果不是，则[R|t]将其转换为.

OpenCV has a good resource on using the camera model (below). Basically, you can project 3D point M' to pixel coordinate m'; this is how you project your vertices to pixel positions. Going the other direction, scale is unrecoverable -- you get the ray M'/s rather than the point M'. The depth you're looking for is s, which is the 3D point's Z coordinate in the camera frame. If your mesh is in a camera-centric frame (X right, Y down, Z out), R = Identity and t = 0. If it's not, [R|t] transforms it to be.

扩展每个因子可以让我们看到矩阵的组成.

Expanding each factor lets us see the makeup of the matrices.

您在下面建议的代码使用OpenCV的projectPoints函数，该函数实现了上面的公式以及一些失真校准(请参见OpenCV主参考).您必须填充矩阵并将其相乘. projectPoints的另一个示例可用在GitHub 上，我相信在此SO问题中将讨论相同的示例.

The code that you suggested below uses OpenCV's projectPoints function, which implements the above equation plus some distortion calibration (see main OpenCV reference). You have to populate the matrices and it multiplies them. An alternative example for projectPoints is available on GitHub, and I believe this same example is discussed in this SO question.

询问者建议的代码

显然，以下代码可以完成这项工作.我可能需要一些时间 鉴于我的C ++知识几乎为零(我 意识到它已被BTW注释掉了):

Apparently the following code does the job. I may need some time to pick through it given that my C++ knowledge is practically zero (I realise that it is commented out BTW):

       //CString str;
       //cv::Mat CamMatrix(3, 3, CV_64F);
       //cv::Mat distCoeffs(5, 1, CV_64F);
       //m_CamCalib.GetOpenCVInfo(&CamMatrix, &distCoeffs);
       //vector<Point3d> GCP_Points;
       //vector<Point2d> Image_Points;
       //cv::Mat RVecs(3, 3, CV_64F); // rotation matrix
       //cv::Mat TranRVecs(3, 3, CV_64F); // rotation matrix
       //cv::Mat TVecs(3, 1, CV_64F); // translation vector
       //RVecs.at<double>(0, 0) = m_CamPosMtrx.m_pMtrx[0];
       //RVecs.at<double>(1, 0) = m_CamPosMtrx.m_pMtrx[1];
       //RVecs.at<double>(2, 0) = m_CamPosMtrx.m_pMtrx[2];

       //RVecs.at<double>(0, 1) = m_CamPosMtrx.m_pMtrx[4];
       //RVecs.at<double>(1, 1) = m_CamPosMtrx.m_pMtrx[5];
       //RVecs.at<double>(2, 1) = m_CamPosMtrx.m_pMtrx[6];

       //RVecs.at<double>(0, 2) = m_CamPosMtrx.m_pMtrx[8];
       //RVecs.at<double>(1, 2) = m_CamPosMtrx.m_pMtrx[9];
       //RVecs.at<double>(2, 2) = m_CamPosMtrx.m_pMtrx[10];
       //transpose(RVecs, TranRVecs);
       //TVecs.at<double>(0, 0) = 0;
       //TVecs.at<double>(1, 0) = 0;
       //TVecs.at<double>(2, 0) = 0;
       //GCP_Points.push_back(Point3d((x - m_CamPosMtrx.m_pMtrx[12]), (y - m_CamPosMtrx.m_pMtrx[13]), (z - m_CamPosMtrx.m_pMtrx[14])));
       //Image_Points.push_back(Point2d(0, 0));
       //projectPoints(GCP_Points, TranRVecs, TVecs, CamMatrix, distCoeffs, Image_Points);

/bool CCameraCalibration::GetOpenCVInfo(Mat * cameraMatrix, Mat * distCoeffs)
//{
//            int i,j;
//            Mat projMatrix;
//            CMatrix4x4 m1;
//            if(cameraMatrix->rows==0) cameraMatrix->create(3,3, CV_64F);
//            if(distCoeffs->rows==0) distCoeffs->create(5, 1, CV_64F);
//            for(i=0;i<3;i++)
//            for(j=0;j<3;j++){
//                   cameraMatrix->at<double>(i,j)=m_pCameraMatrix[i][j];
//            }
//            for(i=0;i<5;i++)
//                   distCoeffs->at<double>(i,0)=m_pCoefficients[i];
//     return false;
//}

这篇关于使用OpenCV和Matlab从校准的图像和三角形网格生成深度图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！