来自两个校准相机的 3D 重建——这个管道中的错误在哪里? [英] 3D reconstruction from two calibrated cameras - where is the error in this pipeline?

问题描述

有很多关于从已知内部校准的立体视图进行 3D 重建的帖子，其中一些是 优秀.我已经阅读了其中的 很多，并且根据我所阅读的内容，我正在尝试使用以下管道/算法计算我自己的 3D 场景重建.我会列出方法，然后在底部提出具体问题.

There are many posts about 3D reconstruction from stereo views of known internal calibration, some of which are excellent. I have read a lot of them, and based on what I have read I am trying to compute my own 3D scene reconstruction with the below pipeline / algorithm. I'll set out the method then ask specific questions at the bottom.

<强>0.校准您的相机:

• 这意味着检索相机 1 和相机 2 的相机校准矩阵 K1 和 K2.这些是封装每个相机的 3x3 矩阵内部参数:焦距、主点偏移/图像中心.这些不会改变，您应该只需要对每个摄像机执行一次，只要您不缩放或更改记录的分辨率即可.
• 离线执行此操作.不要争论.
• 我正在使用 OpenCV 的 CalibrateCamera() 和棋盘例程，但此功能也包含在 Matlab 相机校准工具箱.OpenCV 例程似乎运行良好.

• This means retrieve the camera calibration matrices K₁ and K₂ for Camera 1 and Camera 2. These are 3x3 matrices encapsulating each camera's internal parameters: focal length, principal point offset / image centre. These don't change, you should only need to do this once, well, for each camera as long as you don't zoom or change the resolution you record in.
• Do this offline. Do not argue.
• I'm using OpenCV's CalibrateCamera() and checkerboard routines, but this functionality is also included in the Matlab Camera Calibration toolbox. The OpenCV routines seem to work nicely.

1.基本矩阵 F:

• 现在您的相机已设置为立体装置.使用两个图像/视图之间的点对应关系确定该配置的基本矩阵 (3x3).
• 如何获取对应关系取决于您，并且很大程度上取决于场景本身.
• 我正在使用 OpenCV 的 findFundamentalMat() 来获取 F，它提供了许多方法明智的选项(8 点算法、RANSAC、LMEDS).
• 您可以通过将生成的矩阵代入基本矩阵的定义方程来测试它:x'Fx = 0 其中 x' 和 x 是原始图像点对应关系 (x,y) 在齐次坐标 (x, y, 1) 中，三个向量之一被转置，以便乘法有意义.每个对应关系越接近零，F 越符合它的关系.这相当于检查派生的 F 实际从一个图像平面映射到另一个图像平面的程度.使用 8 点算法，我得到了 ~2px 的平均偏转.

• With your cameras now set up as a stereo rig. Determine the fundamental matrix (3x3) of that configuration using point correspondences between the two images/views.
• How you obtain the correspondences is up to you and will depend a lot on the scene itself.
• I am using OpenCV's findFundamentalMat() to get F, which provides a number of options method wise (8-point algorithm, RANSAC, LMEDS).
• You can test the resulting matrix by plugging it into the defining equation of the Fundamental matrix: x'Fx = 0 where x' and x are the raw image point correspondences (x, y) in homogeneous coordinates (x, y, 1) and one of the three-vectors is transposed so that the multiplication makes sense. The nearer to zero for each correspondence, the better F is obeying it's relation. This is equivalent to checking how well the derived F actually maps from one image plane to another. I get an average deflection of ~2px using the 8-point algorithm.

<强>2.基本矩阵 E:

• 直接根据 F 和校准矩阵计算基本矩阵.
• E = K2^TFK1

3.E 上的内部约束:

• E 应遵守某些约束.特别是，如果通过 SVD 分解成 USV.t，那么它的奇异值应该是 = a, a, 0.S 的前两个对角线元素应该相等，第三个元素为零.
• 我很惊讶地看到 这里 如果您在测试时这不是真的为此，您可以选择从先前的分解中制作一个新的基本矩阵，如下所示: E_new = U * diag(1,1,0) * Vt 这当然可以保证遵守约束.您实际上已经人为地设置了 S = (100,010,000).

• E should obey certain constraints. In particular, if decomposed by SVD into USV.t then it's singular values should be = a, a, 0. The first two diagonal elements of S should be equal, and the third zero.
• I was surprised to read here that if this is not true when you test for it, you might choose to fabricate a new Essential matrix from the prior decomposition like so: E_new = U * diag(1,1,0) * V.t which is of course guaranteed to obey the constraint. You have essentially set S = (100,010,000) artificially.

4.全相机投影矩阵:

• 有两个相机投影矩阵 P₁ 和 P₂.它们是 3x4 并遵守 x = PX 关系.此外，P = K[R|t]，因此 K_inv.P = [R|t](相机校准已被移除).
• 第一个矩阵 P₁(不包括校准矩阵 K)可以设置为 [I|0] 然后 P₂(不包括K) 是 R|t
• 根据 E 的分解计算两个相机之间的旋转和平移 R, t.计算 R 有两种可能的方法(U*W*Vt 和 U*Wt*Vt)和两种计算t的方式(±U的第三列)，也就是说Rt有四种组合，只有一种有效.
• 计算所有四种组合，并选择在几何上与重构点在两个摄像头前面的情况相对应的组合.实际上，我通过执行并计算结果 P₂ = [R|t] 并在归一化坐标中对一些对应的 3d 位置进行三角测量，以确保它们具有正深度(z 坐标)

• There are two camera projection matrices P₁ and P₂. These are 3x4 and obey the x = PX relation. Also, P = K[R|t] and therefore K_inv.P = [R|t] (where the camera calibration has been removed).
• The first matrix P₁ (excluding the calibration matrix K) can be set to [I|0] then P₂ (excluding K) is R|t
• Compute the Rotation and translation between the two cameras R, t from the decomposition of E. There are two possible ways to calculate R (U*W*V.t and U*W.t*V.t) and two ways to calculate t (±third column of U), which means that there are four combinations of Rt, only one of which is valid.
• Compute all four combinations, and choose the one that geometrically corresponds to the situation where a reconstructed point is in front of both cameras. I actually do this by carrying through and calculating the resulting P₂ = [R|t] and triangulating the 3d position of a few correspondences in normalised coordinates to ensure that they have a positive depth (z-coord)

5.3D 三角剖分

• 最后，将恢复的 3x4 投影矩阵与其各自的校准矩阵相结合:P'₁ = K1P₁ 和 P'₂ = K2P₂
• 并相应地对每个 2d 点对应的 3 空间坐标进行三角测量，为此我使用 这里.

• Finally, combine the recovered 3x4 projection matrices with their respective calibration matrices: P'₁ = K₁P₁ and P'₂ = K₂P₂
• And triangulate the 3-space coordinates of each 2d point correspondence accordingly, for which I am using the LinearLS method from here.

问题:

• 此方法是否有遗漏和/或错误?
• 我的 F 矩阵显然是准确的(与典型坐标值相比，映射中的偏差为 0.22%)，但是在使用 normalised 图像针对 x'Ex = 0 测试 E 时该映射中的典型误差大于 100% 的归一化坐标本身.针对 xEx = 0 测试 E 是否有效，如果有效，错误跳转来自哪里?
• 使用 RANSAC 时，我的基本矩阵估计的误差比 8pt 算法差得多，在 x 和 x' 之间的映射中为 ±50px.这让我深感担忧.
• 执行内部约束"对我来说仍然很奇怪 - 仅仅从原始矩阵的部分分解中制造一个新的基本矩阵怎么可能有效?
• 有没有比计算 P 和对一些归一化坐标进行三角剖分更有效的方法来确定要使用 R 和 t 的哪个组合?
• 我的最终重投影错误是 720p 图像中的数百个像素.我是否可能会在 P 矩阵的校准、确定或三角测量中遇到问题?

• Are there any howling omissions and/or errors in this method?
• My F matrix is apparently accurate (0.22% deflection in the mapping compared to typical coordinate values), but when testing E against x'Ex = 0 using normalised image correspondences the typical error in that mapping is >100% of the normalised coordinates themselves. Is testing E against xEx = 0 valid, and if so where is that jump in error coming from?
• The error in my fundamental matrix estimation is significantly worse when using RANSAC than the 8pt algorithm, ±50px in the mapping between x and x'. This deeply concerns me.
• 'Enforcing the internal constraint' still sits very weirdly with me - how can it be valid to just manufacture a new Essential matrix from part of the decomposition of the original?
• Is there a more efficient way of determining which combo of R and t to use than calculating P and triangulating some of the normalised coordinates?
• My final re-projection error is hundreds of pixels in 720p images. Am I likely looking at problems in the calibration, determination of P-matrices or the triangulation?

推荐答案

我的基本矩阵估计中的错误明显更糟使用RANSAC比8pt算法时，在±50px之间的映射x 和 x'.这让我深感担忧.

The error in my fundamental matr1ix estimation is significantly worse when using RANSAC than the 8pt algorithm, ±50px in the mapping between x and x'. This deeply concerns me.

使用8pt算法不排除使用RANSAC原理.直接使用8pt算法的时候，你用的是哪些点呢?你必须自己选择8(好)分.

Using the 8pt algorithm does not exclude using the RANSAC principle. When using the 8pt algorithm directly which points do you use? You have to choose 8 (good) points by yourself.

理论上，您可以从任何点对应关系计算基本矩阵，并且您经常会得到退化的基本矩阵，因为线性方程不是独立的.另一点是 8pt 算法使用超定线性方程组，因此单个异常值将破坏基本矩阵.

In theory you can compute a fundamental matrix from any point correspondences and you often get a degenerated fundamental matrix because the linear equations are not independend. Another point is that the 8pt algorithm uses a overdetermined system of linear equations so that one single outlier will destroy the fundamental matrix.

您是否尝试过使用 RANSAC 结果?我敢打赌它代表了 F 的正确解决方案之一.

Have you tried to use the RANSAC result? I bet it represents one of the correct solutions for F.

我的 F 矩阵显然是准确的(映射中的偏差为 0.22%与典型坐标值相比)，但在测试 E 时x'Ex = 0 使用归一化图像对应关系中的典型错误该映射大于 100% 的归一化坐标本身.是针对 xEx = 0 测试 E 是否有效，如果是，那么错误跳转在哪里来自哪里?

My F matrix is apparently accurate (0.22% deflection in the mapping compared to typical coordinate values), but when testing E against x'Ex = 0 using normalised image correspondences the typical error in that mapping is >100% of the normalised coordinates themselves. Is testing E against xEx = 0 valid, and if so where is that jump in error coming from?

同样，如果 F 是退化的，x'Fx = 0 可以是每个点的对应关系.

Again, if F is degenerated, x'Fx = 0 can be for every point correspondence.

您错误 E 的另一个原因可能是相机的开关(K1T * E * K2 而不是 K2T * E * K1).记得检查:x'Ex = 0

Another reason for you incorrect E may be the switch of the cameras (K1T * E * K2 instead of K2T * E * K1). Remember to check: x'Ex = 0

执行内部约束"对我来说仍然很奇怪——仅仅制造一个新的基本矩阵怎么可能有效部分分解原件?

'Enforcing the internal constraint' still sits very weirdly with me - how can it be valid to just manufacture a new Essential matrix from part of the decomposition of the original?

Hartley 和 Zisserman 在计算机视觉中的多视图几何"中对此进行了解释.据我所知，这与 F 的 Frobenius 范数的最小化有关.

It is explained in 'Multiple View Geometry in Computer Vision' from Hartley and Zisserman. As far as I know it has to do with the minimization of the Frobenius norm of F.

你可以谷歌一下，里面有pdf资源.

You can Google it and there are pdf resources.

有没有更有效的方法来确定 R 和 t 的组合使用比计算 P 和三角化一些归一化的坐标?

Is there a more efficient way of determining which combo of R and t to use than calculating P and triangulating some of the normalised coordinates?

据我所知没有.

我的最终重投影误差是 720p 图像中的数百个像素.是我可能会查看校准中的问题，确定P-矩阵还是三角剖分?

My final re-projection error is hundreds of pixels in 720p images. Am I likely looking at problems in the calibration, determination of P-matrices or the triangulation?

你的刚体变换 P2 不正确，因为 E 不正确.

Your rigid body transformation P2 is incorrect because E is incorrect.

这篇关于来自两个校准相机的 3D 重建——这个管道中的错误在哪里?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！