从两个校准的摄像机进行3D重建-该管道中的错误在哪里? [英] 3D reconstruction from two calibrated cameras - where is the error in this pipeline?

问题描述

关于从已知内部校准的立体视图进行3D重建的文章很多，其中一些是很棒.我已经阅读了其中的很多，并且根据所阅读的内容，我尝试使用以下管道/算法来计算自己的3D场景重构.我将列出方法，然后在底部询问具体问题.

There are many posts about 3D reconstruction from stereo views of known internal calibration, some of which are excellent. I have read a lot of them, and based on what I have read I am trying to compute my own 3D scene reconstruction with the below pipeline / algorithm. I'll set out the method then ask specific questions at the bottom.

0.校准相机:

• 这意味着检索摄像机1和摄像机2的摄像机校准矩阵K ₁ 和K ₂ .这是封装每个摄像机的3x3矩阵内部参数:焦距，主点偏移/图像中心.这些不会改变，只要您不缩放或更改记录的分辨率，您只需为每个摄像机执行一次即可.
• 离线执行此操作.不要争论.
• 我正在使用 OpenCV 的CalibrateCamera()和棋盘格例程，但 Matlab摄像机校准工具箱中也包含此功能. OpenCV例程似乎运行良好.

• This means retrieve the camera calibration matrices K₁ and K₂ for Camera 1 and Camera 2. These are 3x3 matrices encapsulating each camera's internal parameters: focal length, principal point offset / image centre. These don't change, you should only need to do this once, well, for each camera as long as you don't zoom or change the resolution you record in.
• Do this offline. Do not argue.
• I'm using OpenCV's CalibrateCamera() and checkerboard routines, but this functionality is also included in the Matlab Camera Calibration toolbox. The OpenCV routines seem to work nicely.

1.基本矩阵F:

• 现在将您的相机设置为立体声装备.使用两个图像/视图之间的点对应关系确定该配置的基本矩阵(3x3).
• 如何获取对应关系取决于您自己，并且在很大程度上取决于场景本身.
• 我正在使用OpenCV的findFundamentalMat()获取F，F提供了明智的选择方法(8点算法，RANSAC，LMEDS).
• 您可以通过将结果矩阵插入基本矩阵的定义公式中来测试结果矩阵:x'Fx = 0其中，x'和x是均质坐标(x, y, 1)中的原始图像点对应关系(x, y)，并且是以下三个中的一个:向量被转置，以便乘法有意义.每个对应关系越接近零，则F服从它的关系越好.这等效于检查派生的F实际上从一个图像平面映射到另一个图像平面的程度.使用8点算法得到的平均偏移约为2px.

• With your cameras now set up as a stereo rig. Determine the fundamental matrix (3x3) of that configuration using point correspondences between the two images/views.
• How you obtain the correspondences is up to you and will depend a lot on the scene itself.
• I am using OpenCV's findFundamentalMat() to get F, which provides a number of options method wise (8-point algorithm, RANSAC, LMEDS).
• You can test the resulting matrix by plugging it into the defining equation of the Fundamental matrix: x'Fx = 0 where x' and x are the raw image point correspondences (x, y) in homogeneous coordinates (x, y, 1) and one of the three-vectors is transposed so that the multiplication makes sense. The nearer to zero for each correspondence, the better F is obeying it's relation. This is equivalent to checking how well the derived F actually maps from one image plane to another. I get an average deflection of ~2px using the 8-point algorithm.

2.基本矩阵E:

• 直接从F和校准矩阵计算Essential矩阵.
• E = K ₂ ^T FK ₁

• Compute the Essential matrix directly from F and the calibration matrices.
• E = K₂^TFK₁

3.对E的内部约束:

E应该服从某些约束.特别是，如果被SVD分解为USV.t，则其奇异值应为= a, a, 0. S的前两个对角元素应该相等，第三个为零.
• 我很惊讶地阅读此处，如果在测试时这不是真的为此，您可以选择从先前的分解中构造一个新的Essential矩阵，如下所示:E_new = U * diag(1,1,0) * V.t当然可以保证遵守约束.您实质上是人为地设置了S =(100,010,000).

• E should obey certain constraints. In particular, if decomposed by SVD into USV.t then it's singular values should be = a, a, 0. The first two diagonal elements of S should be equal, and the third zero.
• I was surprised to read here that if this is not true when you test for it, you might choose to fabricate a new Essential matrix from the prior decomposition like so: E_new = U * diag(1,1,0) * V.t which is of course guaranteed to obey the constraint. You have essentially set S = (100,010,000) artificially.

4.完整的相机投影矩阵:

• 有两个相机投影矩阵P ₁ 和P ₂ .它们是3x4，并遵守x = PX关系.另外，P = K[R|t]，因此，K_inv.P = [R|t](已取消相机校准的位置).
• 可以将第一个矩阵P ₁ (不包括校准矩阵K)设置为[I|0]，然后将P ₂ (不包括K)设置为R|t
• 根据E的分解来计算两个摄像机R，t之间的旋转和平移.有两种方法可以计算R(U*W*V.t和U*W.t*V.t)，还有两种方法可以计算t(U的±第三列) )，这意味着Rt有四种组合，其中只有一种有效.
• 计算所有四个组合，然后选择在几何上与重构点在两个摄像头前面的情况相对应的一个组合.我实际上是通过计算并计算所得的P ₂ = [R | t]并在标准化坐标中对一些对应关系的3d位置进行三角剖分来确保它们具有正深度(z坐标)

• There are two camera projection matrices P₁ and P₂. These are 3x4 and obey the x = PX relation. Also, P = K[R|t] and therefore K_inv.P = [R|t] (where the camera calibration has been removed).
• The first matrix P₁ (excluding the calibration matrix K) can be set to [I|0] then P₂ (excluding K) is R|t
• Compute the Rotation and translation between the two cameras R, t from the decomposition of E. There are two possible ways to calculate R (U*W*V.t and U*W.t*V.t) and two ways to calculate t (±third column of U), which means that there are four combinations of Rt, only one of which is valid.
• Compute all four combinations, and choose the one that geometrically corresponds to the situation where a reconstructed point is in front of both cameras. I actually do this by carrying through and calculating the resulting P₂ = [R|t] and triangulating the 3d position of a few correspondences in normalised coordinates to ensure that they have a positive depth (z-coord)

5.在3D中进行三角剖分

• 最后，将恢复的3x4投影矩阵与它们各自的校准矩阵组合:P' ₁ = K ₁ P ₁ 和P' ₂ = K ₂ P ₂
• 然后对每个2d点对应关系的3空间坐标进行三角测量，为此，我正在使用

• Finally, combine the recovered 3x4 projection matrices with their respective calibration matrices: P'₁ = K₁P₁ and P'₂ = K₂P₂
• And triangulate the 3-space coordinates of each 2d point correspondence accordingly, for which I am using the LinearLS method from here.

问题:

• 此方法是否有啸叫遗漏和/或错误?
• 我的F矩阵显然是准确的(与典型坐标值相比，映射中的偏移为0.22％)，但是当使用归一化图像对应项针对x'Ex = 0测试E时，映射中的典型误差为> 100％的标准化坐标本身.对xEx = 0进行测试E是否有效?如果是，那么错误跳转来自何处?
• 使用RANSAC时，我的基本矩阵估计中的错误要比8pt算法严重得多，x和x'之间的映射为±50px.这深深地困扰着我.
• 强制执行内部约束"仍然让我感到很奇怪-从原始分解的一部分中仅制造一个新的Essential矩阵如何有效?
• 确定使用R和t的哪个组合比计算P和对某些归一化坐标进行三角测量是否更有效?
• 我最终的重新投影错误是720p图像中的数百个像素.我是否可能在校准，P矩阵确定或三角剖分方面遇到问题?

• Are there any howling omissions and/or errors in this method?
• My F matrix is apparently accurate (0.22% deflection in the mapping compared to typical coordinate values), but when testing E against x'Ex = 0 using normalised image correspondences the typical error in that mapping is >100% of the normalised coordinates themselves. Is testing E against xEx = 0 valid, and if so where is that jump in error coming from?
• The error in my fundamental matrix estimation is significantly worse when using RANSAC than the 8pt algorithm, ±50px in the mapping between x and x'. This deeply concerns me.
• 'Enforcing the internal constraint' still sits very weirdly with me - how can it be valid to just manufacture a new Essential matrix from part of the decomposition of the original?
• Is there a more efficient way of determining which combo of R and t to use than calculating P and triangulating some of the normalised coordinates?
• My final re-projection error is hundreds of pixels in 720p images. Am I likely looking at problems in the calibration, determination of P-matrices or the triangulation?

推荐答案

我的基本matr1ix估计中的错误严重得多 当使用RANSAC而不是8pt算法时，之间的映射为±50px x和x'.这深深地困扰着我.

The error in my fundamental matr1ix estimation is significantly worse when using RANSAC than the 8pt algorithm, ±50px in the mapping between x and x'. This deeply concerns me.

使用8pt算法并不排除使用RANSAC原理. 直接使用8pt算法时，您要使用哪些点?您必须自己选择8(好)点.

Using the 8pt algorithm does not exclude using the RANSAC principle. When using the 8pt algorithm directly which points do you use? You have to choose 8 (good) points by yourself.

理论上，您可以从任何点对应关系计算基本矩阵，并且由于线性方程式不是独立的，因此经常会得到退化的基本矩阵.另一点是，8pt算法使用了一个超定线性方程组，因此一个异常值将破坏基本矩阵.

In theory you can compute a fundamental matrix from any point correspondences and you often get a degenerated fundamental matrix because the linear equations are not independend. Another point is that the 8pt algorithm uses a overdetermined system of linear equations so that one single outlier will destroy the fundamental matrix.

您是否尝试过使用RANSAC结果?我敢打赌，它代表了F的正确解决方案之一.

Have you tried to use the RANSAC result? I bet it represents one of the correct solutions for F.

我的F矩阵显然是准确的(映射中的偏移为0.22％ 与典型坐标值进行比较)，但在对E进行测试时 使用归一化图像对应x'Ex = 0时的典型误差 映射本身是规范化坐标的100％以上.是 针对xEx = 0测试E是否有效，如果是，则该跳转在哪里出错 来自哪里?

My F matrix is apparently accurate (0.22% deflection in the mapping compared to typical coordinate values), but when testing E against x'Ex = 0 using normalised image correspondences the typical error in that mapping is >100% of the normalised coordinates themselves. Is testing E against xEx = 0 valid, and if so where is that jump in error coming from?

同样，如果F退化，则x' F x = 0可以表示每个点对应.

Again, if F is degenerated, x'Fx = 0 can be for every point correspondence.

E错误的另一个原因可能是相机的开关(K1T * E * K2而不是K2T * E * K1).请记住检查:x'Ex = 0

Another reason for you incorrect E may be the switch of the cameras (K1T * E * K2 instead of K2T * E * K1). Remember to check: x'Ex = 0

执行内部约束"仍然让我感到很奇怪- 从中制造一个新的基本矩阵如何有效? 原始分解的一部分?

'Enforcing the internal constraint' still sits very weirdly with me - how can it be valid to just manufacture a new Essential matrix from part of the decomposition of the original?

Hartley和Zisserman撰写的计算机视觉中的多视图几何"中对此进行了解释.据我所知，这与F的Frobenius范数的最小化有关.

It is explained in 'Multiple View Geometry in Computer Vision' from Hartley and Zisserman. As far as I know it has to do with the minimization of the Frobenius norm of F.

您可以对其进行搜索，并且有pdf资源.

You can Google it and there are pdf resources.

是否有更有效的方法来确定R和t的组合 比计算P和对某些归一化的三角剖分使用 坐标?

Is there a more efficient way of determining which combo of R and t to use than calculating P and triangulating some of the normalised coordinates?

据我所知

我最终的重新投影错误是720p图像中的数百个像素.是 我可能会在校准，确定 P矩阵还是三角剖分?

My final re-projection error is hundreds of pixels in 720p images. Am I likely looking at problems in the calibration, determination of P-matrices or the triangulation?

您的刚体变换P2不正确，因为E不正确.

Your rigid body transformation P2 is incorrect because E is incorrect.

这篇关于从两个校准的摄像机进行3D重建-该管道中的错误在哪里?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！