C++/OpenCV - 用于视频稳定的卡尔曼滤波器 [英] C++/OpenCV - Kalman filter for video stabilization

问题描述

我尝试使用卡尔曼滤波器稳定视频以进行平滑处理.但是我有一些问题

每次，我都有两帧:一个是当前帧，另一个是当前帧.这是我的工作流程:

• 计算 goodFeaturesToTrack()
• 使用 calcOpticalFlowPyrLK() 计算光流
• 只保留优点
• 估计刚性变换
• 使用卡尔曼滤波器进行平滑
• 图片变形.

但我认为卡尔曼有问题，因为最后我的视频仍然不稳定，一点也不流畅，甚至比原来的还要糟糕......

这是我的卡尔曼代码

void StabilizationTestSimple2::init_kalman(double x, double y){KF.statePre.at(0)=x；KF.statePre.at<float>(1)=y；KF.statePre.at(2)=0；KF.statePre.at(3)=0；KF.transitionMatrix = *(Mat_(4,4) <<1,0,1,0, 0,1,0,1, 0,0,1,0, 0,0,0,1);KF.processNoiseCov = *(Mat_(4,4) <<0.2,0,0.2,0, 0,0.2,0,0.2, 0,0,0.3,0,0,0,0,0.3);setIdentity(KF.measurementMatrix);setIdentity(KF.processNoiseCov,Scalar::all(1e-6));setIdentity(KF.measurementNoiseCov,Scalar::all(1e-1));setIdentity(KF.errorCovPost, Scalar::all(.1));}

这里是我如何使用它.我只放了有趣的部分.所有这些代码都在一个 flor 循环中.cornerPrev2 和 cornerCurr2 包含之前检测到的所有特征点(使用 calcOpticalFlowPyrLK())

///转换Mat transformMatrix =estimateRigidTransform(cornersPrev2,cornersCurr2,false);//在极少数情况下，找不到转换.我们将只使用最后一个已知的良好变换.if(transformMatrix.data == NULL) {last_transformationmatrix.copyTo(transformMatrix);}transformMatrix.copyTo(last_transformationmatrix);//分解Tdouble dx = transformMatrix.at(0,2);double dy = transformMatrix.at(1,2);double da = atan2(transformMatrix.at(1,0), transformMatrix.at(0,0));//累积帧到帧变换x += dx;y += dy;a += da;std::cout <<"累计 x,y: (" << x << "," << y << ")" <<结束;如果(计算机== 0){init_kalman(x,y);}别的 {矢量<Point2f>平滑特征点；Point2f smooth_feature=kalman_predict_correct(x,y);smooth_feature_point.push_back(smooth_feature);std::cout <<"smooth x,y: (" << smooth_feature.x << "," << smooth_feature.y << ")" <<结束;//目标 - 当前double diff_x = smooth_feature.x - x;//double diff_y = smooth_feature.y - y;dx = dx + diff_x;dy = dy + diff_y;transformMatrix.at(0,0) = cos(da);transformMatrix.at(0,1) = -sin(da);transformMatrix.at(1,0) = sin(da);transformMatrix.at(1,1) = cos(da);transformMatrix.at(0,2) = dx;transformMatrix.at(1,2) = dy;warpAffine(currFrame,outImg,transformMatrix,prevFrame.size(),INTER_NEAREST|WARP_INVERSE_MAP,BORDER_CONSTANT);namedWindow("稳定");imshow("稳定",outImg);命名窗口(原始")；imshow("Original",originalFrame);}

有人可以知道为什么它不起作用吗?

谢谢

解决方案

KF.transitionMatrix = *(Mat_(4,4) <<1,0,1,0, 0,1,0,1, 0,0,1,0, 0,0,0,1);KF.processNoiseCov = *(Mat_(4,4) <<0.2,0,0.2,0, 0,0.2,0,0.2, 0,0,0.3,0,0,0,0,0.3);setIdentity(KF.measurementMatrix);setIdentity(KF.processNoiseCov,Scalar::all(1e-6));setIdentity(KF.measurementNoiseCov,Scalar::all(1e-1));setIdentity(KF.errorCovPost, Scalar::all(.1));

您的 processNoiseCov 不是对称的，我怀疑您是否有正确的非对角线项.在你更好地理解 KF 之前，我会坚持使用对角矩阵.

另一方面，您似乎立即用带有微小值的 setIdentity 覆盖它，这可能是一个错误.也许你在上面的无效矩阵有问题后补充说?

如果您描述了帧速率和状态单位(米?像素?)，我们可以讨论如何为 processNoiseCov 和 measurementNoiseCov 选择合适的值.

好的，鉴于您的状态是 [ x_pixels, y_pixels, dx_pixels, dy_pixels ] 这里有一些提示:

• 你的测量矩阵是 I 所以你是说你测量的东西和你所在的州完全一样(这有点不常见:通常你只会测量你所在州的一些子集，例如，您在测量中没有对速度的估计).
• 鉴于你的测量矩阵是I，processNoiseCov和measurementNoiseCov的含义很相似，所以我将它们放在一起讨论.您的 processNoiseCov 应该是一个对角矩阵，其中每个项是这些值在帧之间如何变化的方差(标准偏差的平方).例如，如果您的像素偏移在每帧 100 像素内有 68% 的机会(请参阅正态分布)，则位置的对角线条目应为 100 * 100 = 10000(请记住，方差是 stddev 的平方).您需要对速度如何变化进行类似的估计.(高级:应该有共同变化的(非对角线)项，因为速度的变化与位置的变化有关，但你可以不用这个，直到对你有意义为止.我已经在其他答案中解释过).
• processNoiseCov 中的附加协方差每帧都会添加，因此请记住所表示的更改超过 1/25 秒.
• 您的 measurementNoiseCov 具有相同类型的单位(同样，测量矩阵是单位)，但应反映您的测量与不可知的实际真实值相比的准确程度.
• 通常，您可以测量过程和测量值的实际值并计算它们的实际协方差(在 excel、python 或 Matlab 或其他中).
• 您的 errorCovPost 是您的初始不确定性，就像您的每帧加法协方差一样表达，但它描述了您对初始状态正确性的确定程度.
• 在使用 KF 时，获得正确的协方差矩阵是最重要的事情.在设计 KF 时，您总是会花更多的时间来做正确的事情.

I try to Stabilize video with a Kalman filter for smoothing . But i have some problems

Each time, i have two frames: one current and another one. Here my workflow:

• Compute goodFeaturesToTrack()
• Compute Optical Flow using calcOpticalFlowPyrLK()
• Keep only good points
• Estimate a rigid transformation
• Smoothing using Kalman filter
• Warping of the picture.

But i think there is something wrong with Kalman because at the end my video is still not stabilized and it's not smooth at all, it even worse than the original...

Here my code of Kalman

void StabilizationTestSimple2::init_kalman(double x, double y)
{

    KF.statePre.at<float>(0) = x;
    KF.statePre.at<float>(1) = y;
    KF.statePre.at<float>(2) = 0;
    KF.statePre.at<float>(3) = 0;

    KF.transitionMatrix = *(Mat_<float>(4,4) << 1,0,1,0,    0,1,0,1,     0,0,1,0,   0,0,0,1);
    KF.processNoiseCov = *(Mat_<float>(4,4) << 0.2,0,0.2,0,  0,0.2,0,0.2,  0,0,0.3,0,
                           0,0,0,0.3);
    setIdentity(KF.measurementMatrix);
    setIdentity(KF.processNoiseCov,Scalar::all(1e-6));
    setIdentity(KF.measurementNoiseCov,Scalar::all(1e-1));
    setIdentity(KF.errorCovPost, Scalar::all(.1));
}

and here how i use it. I put only the interesting part. All this code is inside a flor loop. cornerPrev2 and cornerCurr2 contains all the features points detected just before (with calcOpticalFlowPyrLK())

    /// Transformation
    Mat transformMatrix = estimateRigidTransform(cornersPrev2,cornersCurr2 ,false);

    // in rare cases no transform is found. We'll just use the last known good transform.
    if(transformMatrix.data == NULL) {
        last_transformationmatrix.copyTo(transformMatrix);
    }

    transformMatrix.copyTo(last_transformationmatrix);

    // decompose T
    double dx = transformMatrix.at<double>(0,2);
    double dy = transformMatrix.at<double>(1,2);
    double da = atan2(transformMatrix.at<double>(1,0), transformMatrix.at<double>(0,0));

    // Accumulated frame to frame transform
    x += dx;
    y += dy;
    a += da;
    std::cout << "accumulated x,y: (" << x << "," << y << ")" << endl;

    if (compteur==0){
        init_kalman(x,y);
    }
    else {


          vector<Point2f> smooth_feature_point;
          Point2f smooth_feature=kalman_predict_correct(x,y);
          smooth_feature_point.push_back(smooth_feature);
          std::cout << "smooth x,y: (" << smooth_feature.x << "," << smooth_feature.y << ")" << endl;

          // target - current
          double diff_x = smooth_feature.x - x;//
          double diff_y = smooth_feature.y - y;

          dx = dx + diff_x;
          dy = dy + diff_y;

          transformMatrix.at<double>(0,0) = cos(da);
          transformMatrix.at<double>(0,1) = -sin(da);
          transformMatrix.at<double>(1,0) = sin(da);
          transformMatrix.at<double>(1,1) = cos(da);
          transformMatrix.at<double>(0,2) = dx;
          transformMatrix.at<double>(1,2) = dy;

          warpAffine(currFrame,outImg,transformMatrix,prevFrame.size(), INTER_NEAREST|WARP_INVERSE_MAP, BORDER_CONSTANT);

          namedWindow("stabilized");
          imshow("stabilized",outImg);
          namedWindow("Original");
          imshow("Original",originalFrame);


    }

Can someone have an idea why it's not working ?

Thank

解决方案

KF.transitionMatrix = *(Mat_<float>(4,4) << 1,0,1,0,    0,1,0,1,     0,0,1,0,   0,0,0,1);
KF.processNoiseCov = *(Mat_<float>(4,4) << 0.2,0,0.2,0,  0,0.2,0,0.2,  0,0,0.3,0,
                       0,0,0,0.3);
setIdentity(KF.measurementMatrix);
setIdentity(KF.processNoiseCov,Scalar::all(1e-6));
setIdentity(KF.measurementNoiseCov,Scalar::all(1e-1));
setIdentity(KF.errorCovPost, Scalar::all(.1));

Your processNoiseCov is not symmetric and I doubt you have the right off-diagonal terms. I would stick with a diagonal matrix there until you understand the KF better.

On the other hand, you appear to immediately overwrite it with that setIdentity with tiny values, which is probably a mistake. Maybe you added that after having problems with the invalid matrix above?

If you describe the framerate and the units of your state (meters? pixels?) we can talk about how to pick good values for processNoiseCov and measurementNoiseCov.

EDIT:

Ok, given that your state is [ x_pixels, y_pixels, dx_pixels, dy_pixels ] here are some tips:

• Your measurement matrix is I so you are saying that you are measuring the exact same things as are in your state (this is somewhat uncommon: often you'd only measure some subset of your state, e.g. you'd have no estimate of velocity in your measurements).
• Given that your measurement matrix is I the meaning of the processNoiseCov and measurementNoiseCov are similar, so I will discuss them together. Your processNoiseCov should be a diagonal matrix where each term is the variance (the square of the standard deviation) of how those values might change between frames. For example, if there is a 68% chance (see normal distribution) that your pixel offsets are within 100 pixels each frame, the diagonal entry for position should be 100 * 100 = 10000 (remember, variance is squared stddev). You need to make a similar estimate for how velocity will change. (Advanced: there should be co-varying (off-diagonal) terms because change in velocity is related to change in position, but you can get by without this until that makes sense to you. I've explained it in other answers).
• Your additive covariance in processNoiseCov is added every frame so keep in mind the changes represented are over 1/25th second.
• Your measurementNoiseCov has the same sort of units (again, measurement matrix is identity) but should reflect how accurate your measurements are compared to the unknowable actual true values.
• Often you can measure actual values for your process and measurements and compute their actual covariance (in excel or python or Matlab or whatever).
• Your errorCovPost is your initial uncertainty, expressed just like your per-frame additive covariance, but it is a description of how certain you are about your initial state being correct.
• Getting those covariance matrices right is the most important thing when using a KF. You will always spend more time getting those right than doing anything else when designing a KF.

这篇关于C++/OpenCV - 用于视频稳定的卡尔曼滤波器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！