使用OpenCV将帧转换为从上面取得的帧 [英] Transform a frame to be as if it was taken from above using OpenCV
问题描述
我正在开发一个使用光流技术估计无人机（quadcopter）位置的项目。我目前有一个使用
但是当我在滚动中以10度运行 rotateFrame（origFrame，processedFrame，cameraMatrix，20 *（M_PI / 180），0,0 ）; 图片正在退出框架窗口：
我得出一个结论，我不得不使用4x4单位矩阵，以便能够得到我想要的。为了找到正确的单应性矩阵，我们需要：
- 3D旋转矩阵
R - 相机校准固有矩阵
A1及其反转矩阵A2/ li>
- 翻译矩阵
T。
我们可以通过乘以轴X，Y，Z的旋转矩阵来组成3D旋转矩阵 R ：
Mat R = RZ * RY * RX
转换并保持它居中，我们需要添加由4x4矩阵给出的翻译，其中 dx = 0; dy = 0; dz = 1 ：
Mat T =（Mat_< double>（4,4） <
1，0，0，dx，
0，1，0，dy，
0,0,1，dz，
0,0,0,1） ;给定所有这些矩阵，我们可以构成我们的单应性矩阵 H ： Mat H = A2 *（T *（R * A1））
对于这个单应性矩阵，我们可以使用OpenCV中的 warpPerspective 。
warpPerspective（input，output，H，input.size（），INTER_LANCZOS4）;
此解决方案的完整性和完整性是完整代码：
void rotateImage（const Mat& input，UMat& output，double roll，double pitch，double yaw，
double dx，double dy，double dz，double f，double cx，double cy）
{
//相机校准Intrinsics矩阵
Mat A2 =（Mat_< double>（3,4）< bf，0，cx，0，
0，f，cy，0，
0,0,1,0）
//倒置摄像机校准内在矩阵
Mat A1 =（Mat_ （4,3）<<
1 / f，0，-cx / f，
0，1 / f，-cy / f，
0，0，0，
0，0，1）
//围绕X，Y和Z轴的旋转矩阵
Mat RX =（Mat_ double（4,4）<
1，0，0，
0，cos（roll），-sin（roll），0，
0，sin（roll），cos（roll），0，
0，0，0，
Mat RY =（Mat_< double>（4,4）<<
cos（pitch），0，sin（pitch），0，
0，1,0,0 ，
-sin（pitch），0，cos（pitch），0，
0,0,0,1）
Mat RZ =（Mat_< double>（4,4）<<
cos（yaw）， - sin（yaw），0,0，
sin （偏航），0,0，
0,0,1,0，
0,0,0,1）;
//转换矩阵
Mat T =（Mat_< double>（4,4）<<
1,0,0，dx，
0,1,0 ，dy，
0，0，1，dz，
0，0，0，1）;
//用（RX，RY，RZ）组合旋转矩阵
Mat R = RZ * RY * RX;
//最终变换矩阵
Mat H = A2 *（T *（R * A1））;
//应用矩阵变换
warpPerspective（input，output，H，input.size（），INTER_LANCZOS4）;
}
结果：
I am working on a project for estimating a UAV (quadcopter) location using optical-flow technique. I currently have a code that is using farneback algorithm from OpenCV. The current code is working fine when the camera is always pointing to the ground.
Now, I want to add support to the case when the camera is not pointing straight down - meaning that the quadcopter now has a pitch / roll / yaw (Euler angles). The quadcopters Euler angles are known and I am searching for a method to compute and apply the transformation needed based on the known current Euler angles. So that the result image will be as if it was taken from above (see image below).
I found methods that calculates the transformation when having 2 sets (source and destination) of 4 corners via findHomography or getPerspectiveTransform functions from OpenCV. But I couldn't find any methods that can do it with knowing only Euler angle (because I don't know the destination image corenrs).
So my question is what method can I use and how in order to transform a frame to be as if it was taken from above using only Euler angles and camera height from ground if necessary?
In order to demonstrate what I need:
The relevant part of my current code is below:
for(;;)
{
Mat m, disp, warp;
vector<Point2f> corners;
// take out frame- still distorted
cap >> origFrame;
// undistort the frame using the calibration parameters
cv::undistort(origFrame, undistortFrame, cameraMatrix, distCoeffs, noArray());
// lower the process effort by transforming the picture to gray
cvtColor(undistortFrame, gray, COLOR_BGR2GRAY);
if( !prevgray.empty() )
{
// calculate flow
calcOpticalFlowFarneback(prevgray, gray, uflow, 0.5, 3/*def 3 */, 10/* def 15*/, 3, 3, 1.2 /* def 1.2*/, 0);
uflow.copyTo(flow);
// get average
calcAvgOpticalFlow(flow, 16, corners);
// calculate range of view - 2*tan(fov/2)*distance
rovX = 2*0.44523*distanceSonar*100; // 2 * tan(48/2) * dist(cm)
rovY = 2*0.32492*distanceSonar*100; // 2 * tan(36/2) * dist(cm)
// calculate final x, y location
location[0] += (currLocation.x/WIDTH_RES)*rovX;
location[1] += (currLocation.y/HEIGHT_RES)*rovY;
}
//break conditions
if(waitKey(1)>=0)
break;
if(end_run)
break;
std::swap(prevgray, gray);
}
UPDATE:
After successfully adding the rotation, I still need my image to be centered (and not to go outside of the frame window as shown below). I guess I need some kind of translation. I want the center of the source image to be at the center of the destination image. How can I add this as well?
The rotation function that works:
void rotateFrame(const Mat &input, Mat &output, Mat &A , double roll, double pitch, double yaw){
Mat Rx = (Mat_<double>(3, 3) <<
1, 0, 0,
0, cos(roll), -sin(roll),
0, sin(roll), cos(roll));
Mat Ry = (Mat_<double>(3, 3) <<
cos(pitch), 0, sin(pitch),
0, 1, 0,
-sin(pitch), 0, cos(pitch));
Mat Rz = (Mat_<double>(3, 3) <<
cos(yaw), -sin(yaw), 0,
sin(yaw), cos(yaw), 0,
0, 0, 1);
Mat R = Rx*Ry*Rz;
Mat trans = A*R*A.inv();
warpPerspective(input, output, trans, input.size());
}
When I run it with rotateFrame(origFrame, processedFrame, cameraMatrix, 0, 0, 0); I get image as expected:
But when I run it with 10 degrees in roll rotateFrame(origFrame, processedFrame, cameraMatrix, 20*(M_PI/180), 0, 0);. The image is getting out of the frame window:
解决方案 I came to a conclusion that I had to use the 4x4 Homography matrix in order to be able to get what I wanted. In order to find the right homography matrix we need:
- 3D Rotation matrix
R.
- Camera calibration intrinsic matrix
A1 and its inverted matrix A2.
- Translation matrix
T.
We can compose the 3D rotation matrix R by multiplying the rotation matrices around axes X,Y,Z:
Mat R = RZ * RY * RX
In order to apply the transformation on the image and keep it centered we need to add translation given by a 4x4 matrix, where dx=0; dy=0; dz=1 :
Mat T = (Mat_<double>(4, 4) <<
1, 0, 0, dx,
0, 1, 0, dy,
0, 0, 1, dz,
0, 0, 0, 1);
Given all these matrices we can compose our homography matrix H:
Mat H = A2 * (T * (R * A1))
With this homography matrix we can then use warpPerspective function from OpenCV to apply the transformation.
warpPerspective(input, output, H, input.size(), INTER_LANCZOS4);
For conclusion and completeness of this solution here is the full code:
void rotateImage(const Mat &input, UMat &output, double roll, double pitch, double yaw,
double dx, double dy, double dz, double f, double cx, double cy)
{
// Camera Calibration Intrinsics Matrix
Mat A2 = (Mat_<double>(3,4) <<
f, 0, cx, 0,
0, f, cy, 0,
0, 0, 1, 0);
// Inverted Camera Calibration Intrinsics Matrix
Mat A1 = (Mat_<double>(4,3) <<
1/f, 0, -cx/f,
0, 1/f, -cy/f,
0, 0, 0,
0, 0, 1);
// Rotation matrices around the X, Y, and Z axis
Mat RX = (Mat_<double>(4, 4) <<
1, 0, 0, 0,
0, cos(roll), -sin(roll), 0,
0, sin(roll), cos(roll), 0,
0, 0, 0, 1);
Mat RY = (Mat_<double>(4, 4) <<
cos(pitch), 0, sin(pitch), 0,
0, 1, 0, 0,
-sin(pitch), 0, cos(pitch), 0,
0, 0, 0, 1);
Mat RZ = (Mat_<double>(4, 4) <<
cos(yaw), -sin(yaw), 0, 0,
sin(yaw), cos(yaw), 0, 0,
0, 0, 1, 0,
0, 0, 0, 1);
// Translation matrix
Mat T = (Mat_<double>(4, 4) <<
1, 0, 0, dx,
0, 1, 0, dy,
0, 0, 1, dz,
0, 0, 0, 1);
// Compose rotation matrix with (RX, RY, RZ)
Mat R = RZ * RY * RX;
// Final transformation matrix
Mat H = A2 * (T * (R * A1));
// Apply matrix transformation
warpPerspective(input, output, H, input.size(), INTER_LANCZOS4);
}
Result:
这篇关于使用OpenCV将帧转换为从上面取得的帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
