OpenCV 旋转(Rodrigues)和平移向量,用于在 Unity3D 中定位 3D 对象 [英] OpenCV rotation (Rodrigues) and translation vectors for positioning 3D object in Unity3D

查看:91
本文介绍了OpenCV 旋转(Rodrigues)和平移向量,用于在 Unity3D 中定位 3D 对象的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在使用用于 Unity3d 的 OpenCV"资产(它与用于 Java 的 OpenCV 包相同,但已转换为用于 Unity3d 的 C#),以便为我的理学硕士论文(计算机科学)创建增强现实应用程序.

I'm using "OpenCV for Unity3d" asset (it's the same OpenCV package for Java but translated to C# for Unity3d) in order to create an Augmented Reality application for my MSc Thesis (Computer Science).

到目前为止,我能够使用 ORB 特征检测器从视频帧中检测对象,并且我还可以使用 OpenCV 的 SolvePnP 方法找到 3D 到 2D 的关系(我也进行了相机校准).通过这种方法,我得到了平移和旋转向量.问题出现在增强阶段,我必须将 3d 对象显示为虚拟对象,并在每一帧更新其位置和旋转.OpenCV 返回罗德里格斯旋转矩阵,但 Unity3d 使用四元数旋转,所以我更新对象的位置和旋转错误,我无法弄清楚如何实现转换论坛(从罗德里格斯到四元数).

So far, I'm able to detect an object from video frames using ORB feature detector and also I can find the 3D-to-2D relation using OpenCV's SolvePnP method (I did the camera calibration as well). From that method I'm getting the Translation and Rotation vectors. The problem occurs at the augmentation stage where I have to show a 3d object as a virtual object and update its position and rotation at each frame. OpenCV returns Rodrigues Rotation matrix, but Unity3d works with Quaternion rotation so I'm updating object's position and rotation wrong and I can't figure it out how to implement the conversion forumla (from Rodrigues to Quaternion).

获取 rvec 和 tvec:

Getting the rvec and tvec:

    Mat rvec = new Mat();
    Mat tvec = new Mat();
    Mat rotationMatrix = new Mat ();

    Calib3d.solvePnP (object_world_corners, scene_flat_corners, CalibrationMatrix, DistortionCoefficientsMatrix, rvec, tvec);
    Calib3d.Rodrigues (rvec, rotationMatrix);

更新虚拟对象的位置:

    Vector3 objPosition = new Vector3 (); 
    objPosition.x = (model.transform.position.x + (float)tvec.get (0, 0)[0]);
    objPosition.y = (model.transform.position.y + (float)tvec.get (1, 0)[0]);
    objPosition.z = (model.transform.position.z - (float)tvec.get (2, 0)[0]);
    model.transform.position = objPosition;

我的 Z 轴有一个减号,因为当您将 OpenCV 转换为 Unty3d 的系统坐标时,您必须反转 Z 轴(我自己检查了系统坐标).

I have a minus sign for the Z axis because when you convert OpenCV's to Unty3d's system coordinate you must invert the Z axis (I checked the system coordinates by myself).

Unity3d 的坐标系(绿色是 Y,红色是 X,蓝色是 Z):

Unity3d's Coordinate System (Green is Y, Red is X and Blue is Z) :

OpenCV 的坐标系:

OpenCV's Coordinate System:

此外,我对旋转矩阵做了同样的事情,并更新了虚拟对象的旋转.

In addition I did the same thing for the rotation matrix and I updated the virtual object's rotation.

p.s 我发现了一个类似的问题,但提出这个问题的人并没有明确发布解决方案.

p.s I found a similar question but the guy who asked for it he did not post clearly the solution.

谢谢!

推荐答案

在 cv::solvePnP 之后你就有了旋转 3x3 矩阵.由于该矩阵是旋转矩阵,因此它既是正交的,又是归一化的.因此,该矩阵的列按从左到右的顺序排列:

You have your rotation 3x3 matrix right after cv::solvePnP. That matrix, since it is a rotation, is both orthogonal and normalized. Thus, columns of that matrix are in order from left to right :

  1. 右向量(在 X 轴上);
  2. 向上向量(在 Y 轴上);
  3. 前向矢量(在 Z 轴上).

OpenCV 使用右手坐标系.坐在相机上沿光轴看,X轴向右,Y轴向下,Z轴向前.

OpenCV uses a right-handed coordinates system. Sitting on camera looking along optical axis, X axis goes right, Y axis goes downward and Z axis goes forward.

您将前向向量 F = (fx, fy, fz) 和向上向量 U = (ux, uy, uz) 传递给 Unity.它们分别是第三列和第二列.无需归一化;它们已经标准化了.

You pass forward vector F = (fx, fy, fz) and up vector U = (ux, uy, uz) to Unity. These are the third and second columns respectively. No need to normalize; they are normalized already.

在 Unity 中,您可以像这样构建四元数:

In Unity, you build your quaternion like this:

Vector3 f; // from OpenCV
Vector3 u; // from OpenCV

// notice that Y coordinates here are inverted to pass from OpenCV right-handed coordinates system to Unity left-handed one
Quaternion rot = Quaternion.LookRotation(new Vector3(f.x, -f.y, f.z), new Vector3(u.x, -u.y, u.z));

仅此而已.希望这会有所帮助!

And that is pretty much it. Hope this helps!

编辑了与职位相关的评论

注意:OpenCV 中的 Z 轴位于相机的光轴上,该光轴穿过靠近中心的图像,但通常不完全位于中心.在您的校准参数中,有 Cx 和 Cy 参数.这些组合是图像空间中从中心到 Z 轴穿过图像的位置的 2D 偏移.要在 2D 背景上准确映射 3D 内容,必须考虑这种变化.

NOTE : Z axis in OpenCV is on camera's optical axis which goes through image near center but not exactly at center in general. Among your calibration parameters, there are Cx and Cy parameters. These combined are the 2D offset in image space from center to where the Z axis goes through image. That shift must be taken into account to map exactly 3D stuff over 2D background.

要在 Unity 中正确定位:

To get proper positioning in Unity:

// STEP 1 : fetch position from OpenCV + basic transformation
Vector3 pos; // from OpenCV
pos = new Vector3(pos.x, -pos.y, pos.z); // right-handed coordinates system (OpenCV) to left-handed one (Unity)

// STEP 2 : set virtual camera's frustrum (Unity) to match physical camera's parameters
Vector2 fparams; // from OpenCV (calibration parameters Fx and Fy = focal lengths in pixels)
Vector2 resolution; // image resolution from OpenCV
float vfov =  2.0f * Mathf.Atan(0.5f * resolution.y / fparams.y) * Mathf.Rad2Deg; // virtual camera (pinhole type) vertical field of view
Camera cam; // TODO get reference one way or another
cam.fieldOfView = vfov;
cam.aspect = resolution.x / resolution.y; // you could set a viewport rect with proper aspect as well... I would prefer the viewport approach

// STEP 3 : shift position to compensate for physical camera's optical axis not going exactly through image center
Vector2 cparams; // from OpenCV (calibration parameters Cx and Cy = optical center shifts from image center in pixels)
Vector3 imageCenter = new Vector3(0.5f, 0.5f, pos.z); // in viewport coordinates
Vector3 opticalCenter = new Vector3(0.5f + cparams.x / resolution.x, 0.5f + cparams.y / resolution.y, pos.z); // in viewport coordinates
pos += cam.ViewportToWorldPoint(imageCenter) - cam.ViewportToWorldPoint(opticalCenter); // position is set as if physical camera's optical axis went exactly through image center

您将从物理相机检索到的图像放在以其前轴为中心的虚拟相机正前方(缩放以适合视锥),然后您就可以在 2D 背景上映射适当的 3D 位置!

You put images retrieved from physical camera right in front of virtual camera centered on its forward axis (scaled to fit frustrum) then you have proper 3D positions mapped over 2D background!

这篇关于OpenCV 旋转(Rodrigues)和平移向量,用于在 Unity3D 中定位 3D 对象的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆