OpenCV旋转(Rodrigues)和转换向量，用于在Unity3D中定位3D对象 [英] OpenCV rotation (Rodrigues) and translation vectors for positioning 3D object in Unity3D
问题描述
我正在使用"OpenCV for Unity3d"资产(与Java的OpenCV包相同，但已转换为Unity 3d的C#)，以便为我的MSc论文(计算机科学)创建增强现实应用程序.
I'm using "OpenCV for Unity3d" asset (it's the same OpenCV package for Java but translated to C# for Unity3d) in order to create an Augmented Reality application for my MSc Thesis (Computer Science).
到目前为止，我已经能够使用ORB特征检测器从视频帧中检测到物体，并且还可以使用OpenCV的SolvePnP方法找到3D到2D关系(我也进行了摄像机校准). 通过这种方法，我得到了平移和旋转向量.问题发生在增强阶段，在此阶段，我必须将3d对象显示为虚拟对象，并在每个帧上更新其位置和旋转. OpenCV返回Rodrigues旋转矩阵，但是Unity3d使用Quaternion旋转，因此我错误地更新了对象的位置和旋转，并且我不知道如何实现转换论坛(从Rodrigues到Quaternion).
So far, I'm able to detect an object from video frames using ORB feature detector and also I can find the 3D-to-2D relation using OpenCV's SolvePnP method (I did the camera calibration as well). From that method I'm getting the Translation and Rotation vectors. The problem occurs at the augmentation stage where I have to show a 3d object as a virtual object and update its position and rotation at each frame. OpenCV returns Rodrigues Rotation matrix, but Unity3d works with Quaternion rotation so I'm updating object's position and rotation wrong and I can't figure it out how to implement the conversion forumla (from Rodrigues to Quaternion).
获取rvec和tvec:
Getting the rvec and tvec:
Mat rvec = new Mat();
Mat tvec = new Mat();
Mat rotationMatrix = new Mat ();
Calib3d.solvePnP (object_world_corners, scene_flat_corners, CalibrationMatrix, DistortionCoefficientsMatrix, rvec, tvec);
Calib3d.Rodrigues (rvec, rotationMatrix);
更新虚拟对象的位置:
Vector3 objPosition = new Vector3 ();
objPosition.x = (model.transform.position.x + (float)tvec.get (0, 0)[0]);
objPosition.y = (model.transform.position.y + (float)tvec.get (1, 0)[0]);
objPosition.z = (model.transform.position.z - (float)tvec.get (2, 0)[0]);
model.transform.position = objPosition;
我在Z轴上有一个负号，因为当您将OpenCV转换为Unty3d的系统坐标时，必须反转Z轴(我自己检查了系统坐标).
I have a minus sign for the Z axis because when you convert OpenCV's to Unty3d's system coordinate you must invert the Z axis (I checked the system coordinates by myself).
Unity3d的坐标系(绿色是Y，红色是X，蓝色是Z):
Unity3d's Coordinate System (Green is Y, Red is X and Blue is Z) :
OpenCV的坐标系:
OpenCV's Coordinate System:
此外，我对旋转矩阵做了同样的事情，并且更新了虚拟对象的旋转.
In addition I did the same thing for the rotation matrix and I updated the virtual object's rotation.
p.s我发现了一个类似的问题，但是提出这个问题的那个人没有清楚地发布解决方案.
p.s I found a similar question but the guy who asked for it he did not post clearly the solution.
谢谢！
推荐答案
在cv :: solvePnP之后，您有了3x3旋转矩阵.该矩阵由于是旋转，因此既是正交也是归一化.因此，该矩阵的列按从左到右的顺序排列:
You have your rotation 3x3 matrix right after cv::solvePnP. That matrix, since it is a rotation, is both orthogonal and normalized. Thus, columns of that matrix are in order from left to right :
- 右向量(在X轴上)；
- 向上向量(在Y轴上)；
- 前进矢量(在Z轴上).
OpenCV使用右手坐标系.坐在相机上沿光轴方向看，X轴向右，Y轴向下，Z轴向前.
OpenCV uses a right-handed coordinates system. Sitting on camera looking along optical axis, X axis goes right, Y axis goes downward and Z axis goes forward.
您将前向矢量F =(fx，fy，fz)和向上矢量U =(ux，uy，uz)传递给Unity.这些分别是第三列和第二列.无需标准化；他们已经标准化了.
You pass forward vector F = (fx, fy, fz) and up vector U = (ux, uy, uz) to Unity. These are the third and second columns respectively. No need to normalize; they are normalized already.
在Unity中，您可以这样构建四元数:
In Unity, you build your quaternion like this:
Vector3 f; // from OpenCV
Vector3 u; // from OpenCV
// notice that Y coordinates here are inverted to pass from OpenCV right-handed coordinates system to Unity left-handed one
Quaternion rot = Quaternion.LookRotation(new Vector3(f.x, -f.y, f.z), new Vector3(u.x, -u.y, u.z));
差不多就可以了.希望这会有所帮助！
And that is pretty much it. Hope this helps!
针对位置相关评论进行了编辑
注意:OpenCV中的Z轴位于相机的光轴上，该光轴穿过图像的中心附近，但通常不完全位于中心.在您的校准参数中，有Cx和Cy参数.这些组合是从中心到Z轴穿过图像的位置在图像空间中的2D偏移.必须考虑到这种转变，才能在2D背景上精确地映射3D内容.
NOTE : Z axis in OpenCV is on camera's optical axis which goes through image near center but not exactly at center in general. Among your calibration parameters, there are Cx and Cy parameters. These combined are the 2D offset in image space from center to where the Z axis goes through image. That shift must be taken into account to map exactly 3D stuff over 2D background.
要在Unity中获得正确的位置，
To get proper positioning in Unity:
// STEP 1 : fetch position from OpenCV + basic transformation
Vector3 pos; // from OpenCV
pos = new Vector3(pos.x, -pos.y, pos.z); // right-handed coordinates system (OpenCV) to left-handed one (Unity)
// STEP 2 : set virtual camera's frustrum (Unity) to match physical camera's parameters
Vector2 fparams; // from OpenCV (calibration parameters Fx and Fy = focal lengths in pixels)
Vector2 resolution; // image resolution from OpenCV
float vfov = 2.0f * Mathf.Atan(0.5f * resolution.y / fparams.y) * Mathf.Rad2Deg; // virtual camera (pinhole type) vertical field of view
Camera cam; // TODO get reference one way or another
cam.fieldOfView = vfov;
cam.aspect = resolution.x / resolution.y; // you could set a viewport rect with proper aspect as well... I would prefer the viewport approach
// STEP 3 : shift position to compensate for physical camera's optical axis not going exactly through image center
Vector2 cparams; // from OpenCV (calibration parameters Cx and Cy = optical center shifts from image center in pixels)
Vector3 imageCenter = new Vector3(0.5f, 0.5f, pos.z); // in viewport coordinates
Vector3 opticalCenter = new Vector3(0.5f + cparams.x / resolution.x, 0.5f + cparams.y / resolution.y, pos.z); // in viewport coordinates
pos += cam.ViewportToWorldPoint(imageCenter) - cam.ViewportToWorldPoint(opticalCenter); // position is set as if physical camera's optical axis went exactly through image center
您将从物理摄像机检索到的图像放在虚拟摄像机的正前方，并以其前轴为中心(缩放以适合截锥体)，然后在2D背景上映射了正确的3D位置！
You put images retrieved from physical camera right in front of virtual camera centered on its forward axis (scaled to fit frustrum) then you have proper 3D positions mapped over 2D background!
这篇关于OpenCV旋转(Rodrigues)和转换向量，用于在Unity3D中定位3D对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
