具有已知内在和外在矩阵的立体视觉3d点计算 [英] stereo vision 3d point calculation with known intrinsic and extrinsic matrix

问题描述

我已经使用两个摄像机的固有摄像机矩阵成功计算了旋转，平移。
我也从左右摄像头获得了校正后的图像。现在，我想知道如何计算一个点的3D坐标，即图像中的一个点。在这里，请参阅绿点。我看了方程式，但是它需要基线，我不知道该如何计算。您能告诉我使用给定信息（R，T和本征矩阵）计算绿点的3d坐标的过程吗？

FYI
1.我也有一个基本矩阵和基本矩阵，以防万一我们需要它们。
2.原始图像尺寸为960 x720。已校正的图像尺寸为925 x 669
3.左侧图像的绿点：（562，185），右侧图像的绿点：（542，185）

解决方案

基准一词通常仅表示翻译。由于您已经有了旋转，平移和内在矩阵（请不要使用 R，T 和 K ）。您可以进行三角剖分，并且不需要基本矩阵或基本矩阵（它们可以用于提取 R，T 等，但您已经有了它们）。您也不需要校正图像，因为它不会对三角剖分过程产生太大的影响。有许多种三角测量的方法，每种都有其优缺点，并且有许多实现它们的库。因此，我在这里所做的只是为您提供有关问题和潜在解决方案的概述，以及指向资源的指针，您可以直接使用这些资源，也可以将其用作编写自己的代码的灵感之源。

• 形式化和解决方案概述。让我们在这里正式说明一下。您有一个3d点 X ，有两个观测值 x_1 和 x_2 分别在左侧和右侧图像中。如果对它们进行反向投影，则会获得两条射线：

    ray_1 = K ^ {1} x_1 
   rat_2 = R * K ^ {-1} x_2 + T //我假设[R | T]是第二个摄影机的姿态，以第一个摄影机
    

  理想情况下，您希望这两条光线在 X 点相遇。由于在实践中我们总是会有一些噪声（离散噪声，舍入误差等），两条光线在 X 处不会相遇，所以最好的答案是在 Q 使得

    Q = argmin_X {d（X，ray_1）^ 2 + d（X，ray_2）^ 2} 
    

  其中 d（。）表示线与点之间的欧几里得距离。您可以将其作为常规的最小二乘法解决，也可以采用考虑线段 l 的几何方法（称为 midpoint ）垂直于 ray_1 和 ray_2 ，并以其中间作为解决方案。另一种快速而肮脏的方法是使用DLT。基本上，您将约束重新写成线性系统 ie X 应该尽可能靠近两条光线） > AX = 0 并使用SVD解决。

  
  
  

  通常，几何（中点）方法不太精确。基于DLT的数字虽然不是最稳定的数字，但通常会产生可接受的结果。




• 具有深度形式化的资源

  
  
  

  Hartley-Zisserman's 当然是书！第312页介绍了一种简单的基于DLT的方法，该方法在opencv中使用（在校准和sfm模块中都使用）。该方法非常容易实现，在任何情况下都不应花费超过10分钟语言。

  
  
  

  塞利斯基的书。在有关SFM的一章中对三角剖分进行了有趣的讨论，但没有Hartley-Zisserman的直截了当或深入。




• 代码。 您可以通过calib3d模块或contribs / sfm模块从opencv使用三角剖分方法。两者都使用DLT，但是SFM模块中的代码更容易理解（calib3d代码具有很多老式的C代码，阅读起来并不愉快）。还有另一个库，称为openGV，它具有一些有趣的三角剖分方法。

  
  
  

  cv :: triangulatePoints

  
  
  

  cv :: sfm :: triangulatePoints

  
  
  

  OpenGV

  
  
  

  openGV git repo似乎不太活跃，我也不是库设计的忠实拥护者，但是如果我没记错的话（可以告诉我），它提供了除三角剖分法之外的其他方法。 p>
  
  

  自然地，这些都是用C ++编写的，但是如果您使用其他语言，查找包装器或类似的库将不会很困难（使用python时，您仍然拥有opencv包装器，而MATLAB具有束调制e等）。

I have successfully calculated Rotation, Translation with the intrinsic camera matrix of two cameras. I also got rectified images from the left and right cameras. Now, I wonder how I calculate the 3D coordinate of a point, just one point in an image. Here, please see the green points. I have a look at the equation, but it requires baseline which I don't know how to calculate. Could you show me the process of calculating the 3d coordinate of the green point with the given information (R, T, and intrinsic matrix)?

FYI 1. I also have a Fundamental matrix and Essential matrix, just in case we need them. 2. Original image size is 960 x 720. Rectified ones are 925 x 669 3. The green point from the left image: (562, 185), from the right image: (542, 185)

解决方案

The term "baseline" usually just means translation. Since you already have your rotation, translation and intrinsics matrices (let's not them R, T and K). you can triangulate and don't need either the Fundamental or Essential matrices (they could be used to extract R, T etc but you already have them). You don't really need your images to be rectified either, since it doesn't change the triangulation process that much. There are many ways to triangulate, each with their pros and cons, and many libraries that implement them. So, all I can do here is give you and overview of the problem and potential solutions, as well as pointers to resources that you can either use as their are or as a source of inspiration to write your own code.

• Formalization and solution outlines. Let's formalize what we are after here. You have a 3d point X, with two observations x_1 and x_2 respectively in the left and right images. If you backproject them, you obtain two rays:

  ray_1=K^{1}x_1
  rat_2=R*K^{-1}x_2+T  //I'm assuming that [R|T] is the pose of the second camera expressed in the referential of the first camera 
  

  Ideally, you'd want those two rays to meet at point X. Since in practice we always have some noise (discretization noise, rounding errors and so on) the two rays wont meet at X, so the best answer would be a point Q such that

  Q=argmin_X {d(X,ray_1)^2+d(X,ray_2)^2}
  

  where d(.) denotes the Euclidian distance between a line and a point. You can solve this problem as a regular least squares problem, or you can just take the geometric approach (called midpoint) of considering the line segment l that is perpendicular to both ray_1 and ray_2, and take its middle as your solution. Another quick and dirty way is to use the DLT. Basically, you re-write the constrains (i.e. X should be as close as possible to both rays) as a linear system AX=0 and solve it with SVD.

  Usually, the geometric (midpoint) method is the less precise. The DLT based one, while not the most stable numerically, usually produces acceptable results.

• Ressources that present in depth formalization

  Hartley-Zisserman's book of course! Chapter 12. A simple DLT-based method, which is the one used in opencv (both in the calibration and sfm modules) is explained on page 312. It is very easy to implement, it shouldn't take more that 10 minutes in any language.

  Szeliski'st book. It has an intersting discussion on triangulation in the chapter on SFM, but is not as straight-forward or in depth as Hartley-Zisserman's.

• Code. You can use the triangulation methods from opencv, either from the calib3d module, or from the contribs/sfm module. Both use the DLT, but the code from the SFM module is more easily understandable (the calib3d code has a lot of old-school C code which is not very pleasant to read). There is also another lib, called openGV, which has a few interesting methods for triangulation.

  cv::triangulatePoints

  cv::sfm::triangulatePoints

  OpenGV

  The openGV git repo doesn't seem very active, and I'm not a big fan of the design of the library, but if I remember correctly (feel free to tell me otherwise) it offers methods other that the DLT for triangulations.

  Naturally, those are all written in C++, but if you use other languages, finding wrappers or similar libraries wont be difficult (with python you still have opencv wrappers, and MATLAB has a bundle module, etc.).

这篇关于具有已知内在和外在矩阵的立体视觉3d点计算的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！