使用opencv确定外在相机与世界空间对象的opengl [英] Determine extrinsic camera with opencv to opengl with world space object

问题描述

我正在使用opencv和openframeworks(即opengl)从图像(以及随后用于三角测量的几幅图像)中计算摄像机(世界变换和投影矩阵).

I'm using opencv and openframeworks (ie. opengl) to calculate a camera (world transform and projection matrixes) from an image (and later, several images for triangulation).

出于opencv的目的，平面图"成为对象(即棋盘)，其世界中心为0,0,0. 世界/楼层的位置是已知的，所以我需要获取投影信息(失真系数，fov等)和摄像机的外部坐标.

For the purposes of opencv, the "floor plan" becomes the object (ie. the chessboard) with 0,0,0 the center of the world. The world/floor positions are known so I need to get the projection information (distortion coefficients, fov etc) and the extrinsic coordinates of the camera.

我已将这些平面图点的视图位置映射到2D图像的归一化视图空间中([0,0]位于左上角，[1,1]位于右下角).

I have mapped the view-positions of these floor-plan points onto my 2D image in normalised view-space ([0,0] is top-left. [1,1] is bottom right).

对象(平面图/世界点)在xz平面上，-y向上，因此我将opencv转换为xy平面(此处不确定z-up为负还是正...)，因为它需要平面

The Object (floor plan/world points) is on xz plane, -y up, so I convert to the xy plane (Not sure here if z-up is negative or positive...) for opencv as it needs to be planar

ofMatrix4x4 gWorldToCalibration(
    1, 0, 0, 0,
    0, 0, 1, 0,
    0, 1, 0, 0,
    0, 0, 0, 1
    );

我将1,1作为ImageSize传递给CalibrateCamera. 标志是CV_CALIB_FIX_ASPECT_RATIO|V_CALIB_FIX_K4|CV_CALIB_FIX_K5 calibrateCamera成功运行，给我一个低错误(通常在0.003附近).

I pass 1,1 as the ImageSize to calibrateCamera. flags are CV_CALIB_FIX_ASPECT_RATIO|V_CALIB_FIX_K4|CV_CALIB_FIX_K5 calibrateCamera runs successfully, gives me a low-error (usually around 0.003).

使用calibrationMatrixValues我会得到一个合理的FOV，通常在50度左右，所以我很确定其固有属性是正确的.

using calibrationMatrixValues I get a sensible FOV, usually around 50 degrees, so I'm pretty sure the intrinsic properties are correct.

现在要计算摄像机的外部世界空间变换...我不相信我需要使用solvePnP，因为我只有一个对象(尽管我之前尝试过使用它并返回了相同的结果)

Now to calculate the extrinsic world-space transform of the camera... I don't believe I need to use solvePnP as I only have one object (although I tried all this before with it and came back with the same results)

//  rot and trans output...
cv::Mat& RotationVector = ObjectRotations[0];
cv::Mat& TranslationVector = ObjectTranslations[0];

//  convert rotation to matrix
cv::Mat expandedRotationVector;
cv::Rodrigues(RotationVector, expandedRotationVector);

//  merge translation and rotation into a model-view matrix
cv::Mat Rt = cv::Mat::zeros(4, 4, CV_64FC1);
for (int y = 0; y < 3; y++)
   for (int x = 0; x < 3; x++) 
        Rt.at<double>(y, x) = expandedRotationVector.at<double>(y, x);
Rt.at<double>(0, 3) = TranslationVector.at<double>(0, 0);
Rt.at<double>(1, 3) = TranslationVector.at<double>(1, 0);
Rt.at<double>(2, 3) = TranslationVector.at<double>(2, 0);
Rt.at<double>(3, 3) = 1.0;

现在我可以轮换了&转换矩阵，但它是专栏的(我相信，如果不进行转置，对象将完全歪斜，并且上面的代码对我来说是专栏的)

Now I've got a rotation & transform matrix, but it's column major (I believe as the object is totally skewed if I don't transpose, and the code above looks column major to me)

//  convert to openframeworks matrix AND transpose at the same time
ofMatrix4x4 ModelView;
for ( int r=0;  r<4;    r++ )
    for ( int c=0;  c<4;    c++ )
        ModelView(r,c) = Rt.at<double>( c, r ); 

使用之前矩阵的逆函数将飞机交换回我的坐标空间(y向上).

Swap my planes back to my-coordinate space (y up) using the inverse of the matrix before.

//  swap y & z planes so y is up
ofMatrix4x4 gCalibrationToWorld = gWorldToCalibration.getInverse();
ModelView *= gCalibrationToWorld;

不确定我是否需要这样做...当我将飞机放入校准中时，我并没有否定飞机...

Not sure if I NEED to do this... I didn't negate the planes when I put them INTO the calibration...

//  invert y and z planes for -/+ differences between opencv and opengl
ofMatrix4x4 InvertHandednessMatrix(
    1,  0,  0, 0,
    0,  -1, 0, 0,
    0,  0,  -1, 0,
    0,  0,  0,  1
    );
ModelView *= InvertHandednessMatrix;

最后，模型视图是相对于相机的对象，我想将其反转为相对于相机的对象(0,0,0)

And finally, the model view is object-relative-to-camera and I want to invert it to be camera-relative-to-object(0,0,0)

ModelView = ModelView.getInverse();

这会导致相机放置在错误的位置，并且旋转错误.距离不是太远，相机在Y平面的右侧，平移也不是疯狂的，我认为这是正确的方向....尚不正确. 涂有油漆的蓝色圆圈是我希望相机放置的位置.

This results in a camera in the wrong place, and rotated wrong. It's not too far off, the camera is the right side of the Y plane, the translation isn't wildly off, and I think it's the right way up.... just not correct yet. The paint-drawn blue circle is where I expect the camera to be.

我已经遍历了数十次SO解答和文档，但是还没有找到正确的方法，我敢肯定我已经满足了空间转换方面的所有需求，但是也许我已经遗漏了一些明显的东西? 还是做错了顺序?

I've gone through loads of SO answers, the documentation a dozen times, but not quite found anything right, I'm pretty sure I've covered everything I need to space-conversion-wise, but maybe I've missed something obvious? Or doing something in the wrong order?

更新1-世界空间飞机... 我已将世界空间平面更改为XY(Z向上)以匹配openCV的输入. (gWorldToCalibration现在是一个单位矩阵). 旋转仍然是错误的，并且投影输出是相同的，但是我认为现在的翻译是正确的(肯定在标记的正确一侧)

Update 1 - world-space plane... I've changed my world-space floor plane to XY(Z up) to match the input for openCV. (gWorldToCalibration is now an identity matrix). The rotation is still wrong, and the projection output is the same, but I think the translation is correct now (It's certainly on the correct side of the markers)

Update2-实际图片大小 我正在将图像尺寸用于相机校准.看到我正在使用已标准化的1,1，但是imageSize参数是整数，我认为这可能很重要...而且我猜是(红色框是投影的视图空间点与z相交的地方) = 0楼板) 没有任何失真校正，这是结果(仅更改的是图像大小从1,1到640,480.我也将归一化的输入视图空间坐标乘以640,480) 我将尝试添加失真校正，以查看它是否完美地对齐 ...

Update2 - Real image size I'm playing with the image size going into the camera calibration; seeing as I'm using 1,1 which is normalised, but the imageSize parameter is in integers, I thought this might be significant... And I guess it is (The red box is where the projected view-space points intersect with z=0 floor plane) Without any distortion correction, here is the result (Only thing changed is imagesize from 1,1 to 640,480. I multiply my normalised input-view-space coords by 640,480 too) I'm going to try and add distortion correction to see if it lines up perfectly...

推荐答案

到目前为止，至少我将Edit 2(ImageSize必须大于1,1)视为修复程序，因为它产生的结果要多得多就像我期望的那样.

For now at least I'm treating my Edit 2 (ImageSize must be something bigger than 1,1) as the fix as it produced results much, much more like what I was expecting.

此刻我可能会倒挂东西，但这产生了很好的结果.

I might have things upside down at the moment, but this is producing pretty good results.

这篇关于使用opencv确定外在相机与世界空间对象的opengl的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！