我如何估计相机姿势与3d到2d点对应（使用opencv） [英] How can I estimate the camera pose with 3d-to-2d-point-correspondences (using opencv)
问题描述
您的目标是开发在飞机（模拟器）驾驶舱中使用的头部跟踪功能，以便提供AR以支持民用飞行员登陆和飞行与视觉条件恶劣。
我的方法是检测特征点（在黑暗的模拟器LED），我知道它的3D坐标，而不是计算估计的（头戴式相机）姿势[R | t]（旋转与平移并置）。
我所遇到的问题是，估计姿势似乎总是错误，并且投影了我的3D点（我也习惯了估计姿势）与2D图像点不重叠（或不可见）。
我的问题是：
如何通过一组给定的2D到3D点对应来估计摄像机姿势。
为什么它无法正常工作，以及我在哪里可能是错误的原因？
测量（3D和2D点和相机矩阵），以获得在现实生活环境中工作的理论解决方案？
我使用的硬件是爱普生BT-200。
在飞机中我定义了一个固定的纵坐标，我希望相对平移和旋转是我的程序的结果。程序检测（唯一）LED的图像坐标，并将它们与其对应的3D坐标匹配。使用open cv示例android代码（的不同求解算法标志和不同参数。
也许你可以以某种方式帮助我找到我的错误，或者告诉我一个好的路径来解决我的初始问题。我将附加我的调试源代码与许多println语句和调试图像。 此代码包含我的测量点。
感谢您的帮助。
类应提供3个解决方案，但OpenCV只提供一个。 EPNP应该返回良好的结果，但通过 EPNP OpenCV返回最差的结果，从我的人的obersation评价。
现在的问题是，如何过滤不准确的值或确保OpenCV函数返回有效值。 （可能我修改本机代码以接收PnP的3个解决方案）。
压缩图片 （37MB），显示我的当前结果（使用IterATIVE PnP-Solver），内在猜测为零旋转和向上75厘米。打印输出有x轴向前，y轴向左和z向下，以及腐蚀性的侧倾，俯仰和偏航角。
我试图实现我的头部跟踪系统学到的一件事是，你应该从简单的问题，而不是移动到更复杂的一个。你的问题是相当昂贵，不幸的是我没有时间来分析它，并在代码中搜索错误或逻辑错误，所以至少我会尝试给你一些提示和工作示例。
这里是OpenCV教程，用于查找对象的平移和旋转。它是用Python编写的，如果这是一个问题，这里部分我的旧c ++项目。
我的项目使用solvePnP或solvePnPRansac函数执行相同的任务（您可以更改模式）。注意我的代码它是一些旧的操场项目的一部分，所以即使在清除后，我执行它是相当混乱。当你运行它，显示打印的棋盘到相机，按'p'开始位置和旋转估计，'m'改变模式（0- ransac，1-pnp，2-posit似乎不工作...）或'd'使用分裂系数打开/关闭。
这两个项目都依赖于找到棋盘模式，但它很容易修改它们以使用一些其他对象。
相机校准 - 我一直在我的头部跟踪系统，我从来没有设法校准相机两次相同的结果...所以我决定使用一些calibartion文件，我发现在github上，它工作得很好 - 在这里，你可以找到一个更多的信息有关这个文件的链接。
编辑
尽可能简单的解决方案，在一些（甚至简单）的情况下给出良好的结果。我认为开始的一个好点是，从教学课程（这一个），使其工作。从这开始你的问题将比开始你的问题容易得多。尝试使用任何编程语言的任何工作解决方案 - 考虑使用Python或C ++版本的OpenCV - 有更多的教程/示例比Java版本和比较结果从您的代码与一些工作代码的结果将使更容易。当您将有一些有效的解决方案尝试修改它与您的测试环境一起使用。有很多事情，可能会导致它不能正常工作 - 没有足够的点，错误的代码，甚至在OpenCV的Java包装器，结果的错误解释等...
edit2：
使用您的代码中的点数， / p>
tvec = [[ 36.23910413]，[-82.83704819]，[266.03157578]]
不幸的是，对我来说很难说结果是否好。 ..对我来说唯一可能是错误的是2个角度不同于0（或180）。但如果你从（355,37），（353,72），（353,101）更改 points2d 的最后一行，到
（355,37），（35 5 ，72） / strong>，101）
（我猜这是你的错误，不是正确的结果） b
$ b
rvec = [[-159.34101842]，[1.04951033]，[-11.43731376]]
tvec = [[-25.74308282] -82.58461674]，[268.12321097]]
这可能更接近正确的结果。更改相机矩阵更改会产生很多结果，因此请考虑从此帖中测试值。
请注意，所有rvec值都乘以 180.0 / 3.14 - 在c ++和python rvec向量由solvePnPRansac返回以弧度表示的角度。
Hello my goal is to develop head-tracking functionality to be used in an aircraft (simulator) cockpit, in order to provide AR to suport civilian pilots to land and fly with bad visual conditions.
My approach is to detect characteristic points (in the dark simulator LEDs) of which I know the 3D coordinates and than compute the estimated (head worn camera's) pose [R|t] (rotation concatinated with translation).
The problem I do have is that the estimated pose seems to be always wrong and a projection of my 3D points (which I also used to estimate the pose) does not overlap with the 2D image points (or is not visible).
My questions are:
How can I estimate the camera pose with a given set of 2D-to-3D point correspondences.
Why does it not work how I try it and where might be sources of error?
How accurate must be the measurements (of 3D and 2D points and the camera matrix) to get the theoretical solution working in a real life environment?
Will the approach work for coplanar points (x,y axis changed) in theory?
The hardware I use is the Epson BT-200.
In the aircraft I defined a fixed ordinate to which I expect relative translations and rotations as result of my program. The program detects the image coordinates of (unique) LEDs and matches them to their corresponding 3D coordinate. With a camera matrix I obtained using the open-cv sample android code (https://github.com/Itseez/opencv/tree/master/samples/android/camera-calibration) I try to estimate the pose using solvePnP.
My camera matrix and distortion varries slightly. Here are some values I received from the procedure. I made sure that the circle-distance of my printed out circle pattern is the same as written down in the source-code (measured in Meters).
Here are some examples and how I create the OpenCV Mat of it.
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.04569467373955304,
// 0.1402980385369059,
// 0,
// 0,
// -0.2982135315849994
// };
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.08245931646421553,
// -0.9893762277047577,
// 0,
// 0,
// 3.23553287438898
// };
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.07444480392067945,
// -0.7817175834131075,
// 0,
// 0,
// 2.65433773093283
// };
protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
/*This matrix should have 5 values*/
0.08909941096327206,
-0.9537960457721699,
0,
0,
3.449728790843752
};
protected final double[][] CAMERA_MATRIX_VALUES = new double[][]{
/*This matrix should have 3x3 values*/
// {748.6595405553738, 0, 319.5},
// {0, 748.6595405553738, 239.5},
// {0, 0, 1}
// {698.1744297982436, 0, 320},
// {0, 698.1744297982436, 240},
// {0, 0, 1}
// {707.1226937511951, 0, 319.5},
// {0, 707.1226937511951, 239.5},
// {0, 0, 1}
{702.1458656346429, 0, 319.5},
{0, 702.1458656346429, 239.5},
{0, 0, 1}
};
private void initDestortionMatrix(){
distortionMatrix = new MatOfDouble();
distortionMatrix.fromArray(DISTORTION_MATRIX_VALUES);
}
private void initCameraMatrix(){
cameraMatrix = new Mat(new Size(3,3), CvType.CV_64F);
for(int i=0;i<CAMERA_MATRIX_VALUES.length; i++){
cameraMatrix.put(i, 0, CAMERA_MATRIX_VALUES[i]);
}
}
To estimate the camera pose I do use solvePnP (and solvePnPRansac) as described in several locations (1,2,3,4). The result of solvePnP I use as input for the Projection (Calib3d.projectPoints). The inverse of the concatinated result [R|t] I do use as estimated pose.
Because my results in the productive environment were too bad I created a testing environment. In that environment I place the camera (which is because of it's 3D-shape (it's a glass) slightly rotated downwards at a table's edge. This edge I do use as ordinate of the world-coordinate system. I searched how the open-cv coordinate system might be oriented and found different answers (one on stackoverflow and one in an official youtube-talk about opencv). Anyways I tested if I got the coordinate system right by projection 3D points (described in that coordinate system) on an image and checked if the given world shape stays constant.
So I came up wiht z pointing foreward, y downward and x to the right.
To get closer to my solution I estimated the pose in my testing environment. The translation vector-output and euler angel output refers to the inverse of [R|t]. The euler angels might not be displayed correct (they might be swaped or wrong, if we take order into account) because I compute it with the convetional (I assume refering to the airplane coordinate system) equations, using an open-cv coordinate system. (The computation happens in the class Pose which I will attach). But anyways even the translation vector (of the inverse) appeard to be wrong (in my simple test).
In one test with that Image I had a roll (which might be pitch in airplane coordinates) of 30° and a translation upwards of 50cm. That appeard to be more reasonable. So I assumed because my points are coplanar, I might get ambiguous results. So I realized an other test with a point which changed in the Z-Axis. But with this test even the projection failed.
For solvePnP I tried all different solving-algorithm-flags and different parameters for the ransac algorithm.
Maybe you can somehow help me to find my mistake, or showing me a good path to solve my initial problem. I am going to attach also my debugging source-code with many println statements and the debugging images. This code contains my point measurements.
Thanks for your help in advance.
Class Main.java: Class Pose.java: 0.png
1.png
EDIT 22.03.2015: Finally I have been able to find mistakes I made.
- I modified a Mat object in a for-loop, because OpenCV works a lot with call by reference, and I was not careful enough here. So the tvec and rvec for the reprojection were not right.
- One of my points in the testing environment had (in the image coordinates), was tagged wrong due to an axis-direction confusion.
So my approach in general was right. I am not receiving at least (often) valid reprojections in my test-dataset.
Unfortunately the OpenCV PnP algorithms: "ITERATIVE, P3P, EPNP" return various results, and even with using a very unaccurate but close intrinsic guess, the results are only sometimes correct. The P3P algorithm is supposed to provide 3 solutions, but OpenCV only provides one. EPNP is supposed to return good results, but with EPNP OpenCV returns the worst results, evaluated from my human obersation.
The problem now is, how to filter the inaccurate values or ensure the OpenCV function returns valid ones. (Maybe I shuold modify the native code to receive 3 solutions for PnP).
The compressed images here (37MB), do show my current results (with the ITERATIVE PnP-Solver) , with an intrinsic guess of zero rotation and 75 cm upwards. The print-out has an x-axis foreward, y-axis to the left and z-down, and corrosponding roll, pitch, and yaw angles.
One thing that i've learned during trying to implement my head tracking system is that you should start from simple problem and than move to more complicated one. Your question is quite ong and unfortunetely i don't have time to analyze it and search for a bug or logical mistake in your code, so at least i will try to give you some hints and working examples.
Here is OpenCV tutorial for finding object translation and rotation. It's written in Python, if it is a problem here part of my old c++ project.
My project performs the same task using solvePnP or solvePnPRansac function (you can change mode). Note that my code it's a part of some old "playground" project, so even after cleaining which i performed it's quite messy. When you run it, show printed chessboard to the camera, press 'p' to start position and rotation estimation, 'm' to change mode (0-ransac, 1-pnp, 2-posit which seems to not work...) or 'd' to turn on/off using dissortion coefficients.
Both projects relies on finding chessboard pattern, but it shoud be easy to modify them to use some other objects.
Camera calibration - while i've been working on my head tracking system i've never managed to calibrate camera twice with the same results... So i decided to use some calibartion file which i've found on github and it worked well - here you can found a litte more information about that an link to this file.
edit:
Try to start with as simple as possible solution that gives good results in some (even simple) situation. A good point to start in my opinion is to replace a sheet of paper from your testing environment with printed chessboard from tutorial (this one) and make it working. Moving from this to your problem will be much easier than beginning with you problem. Try to make any working solution in any programming language - consider using Python or C++ version of OpenCV - there is much more tutorials/examples than to Java version and comparing results from your code with results from some working code will make it much easier. When you will have some working solution try to modify it to work with your testing environment. There is a lot of things which may cause it not working right now - not enough points, bug in your code or even in OpenCV Java wrapper, bad interpretation of results, etc...
edit2:
Using points from your code i've managed to get following results:
rvec = [[-158.56293283], [ 1.46777938], [ -17.32569125]]
tvec = [[ -36.23910413], [ -82.83704819], [ 266.03157578]]
Unfortunetely, for me it's hard to say whether results are good or not... The only thing that might be wrong to me is that 2 angles are different from 0 (or 180). But if you change last row of points2d from (355,37), (353,72), (353,101) to
(355,37), (355,72), (355,101)
(i guess it's your mistake, not a correct result) you will get:
rvec = [[-159.34101842], [ 1.04951033], [ -11.43731376]]
tvec = [[ -25.74308282], [ -82.58461674], [ 268.12321097]]
which might be much closer to the correct result. Changing camera matrix changes results much, so consider testing values from this post.
Note that all rvec values are multiplied by 180.0/3.14 - in c++ and python rvec vector returned by solvePnPRansac contains angles in radians.
这篇关于我如何估计相机姿势与3d到2d点对应（使用opencv）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
