解决低FPS的相关代码以计算图像中的偏移 [英] Tackle low FPS for correlation code to compute shift in Image

问题描述

我正在尝试使用相关性来跟踪对象。我正在逐帧显示较大图像中的较小补丁。为此，我正在寻找补丁的变化，并且在相关性最大的地方，使用新补丁更新补丁。

I am trying to track an object using correlation. I am finding a smaller patch in a larger image, frame by frame. For this, I am finding the shift in the patch, and where correlation is maximum, update the patch with a new patch.

我的代码是：

cv::Mat im_float_2,imagePart_out;
cv::Mat im_floatBig;
cv::Scalar im1_Mean, im1_Std, im2_Mean, im2_Std;

double covar, correl;
int n_pixels;

void computeShift()
{
    int maxRow=0, maxCol=0, TX, TY;
    double GMAX=0;
    Mat image_window = Mat::zeros(imagePart.rows, imagePart.cols, CV_32F);

    imagePart.convertTo(im_float_2, CV_32F);
    imageBig.convertTo(im_floatBig,CV_32F);

    for(maxRow=0; maxRow<=imageBig.rows-image_window.rows;maxRow++)
    {
        for(maxCol=0; maxCol<imageBig.cols-image_window.cols;maxCol++)
        {

            image_window = im_floatBig( cv::Rect( maxCol, maxRow, 
            image_window.cols, image_window.rows ) );

            n_pixels = image_window.rows * image_window.cols;

            // Compute mean and standard deviation of both images

            meanStdDev(image_window, im1_Mean, im1_Std);
            meanStdDev(im_float_2, im2_Mean, im2_Std);

            // Compute covariance and correlation coefficient
            covar = (image_window - im1_Mean).dot(im_float_2 - im2_Mean) / n_pixels;

            correl = covar / (im1_Std[0] * im2_Std[0]);
            if (correl > GMAX)
            {
            GMAX = correl; TX = maxRow; TY=maxCol;
            image_window.convertTo(imagePart, CV_8UC1);
            }
        }
    }

            cvtColor(imagePart, imagePart_out, CV_GRAY2BGR);
            printf("\nComputed shift: [%d, %d] MAX: %f\n", TX, TY,GMAX);                

}

但是执行此操作时，我得到的FPS非常低（1 -2），即使是较小的视频尺寸（帧尺寸- 262x240 ，色块尺寸- 25x25 ）。

But when executing this I am getting very low FPS(1-2) even for small video size (Frame size-262x240, Patch size- 25x25).

有没有办法实现更高的FPS。我也在寻找相位相关的方向，但不确定如何从这里开始。

Is there any way to achieve higher FPS. I am also looking in the direction of phase correlation, but not sure how to go about it from here. Can converting it to frequency domain will help?

现在，我想优化上面的代码以提高速度。

For now, I want to optimize the above code for speed.

推荐答案

是的，使用FFT可能会有所帮助。只需将 im_float_2 填充到 im_floatBig 的大小即可。取其中一个变换的复共轭后，在傅立叶域中相乘会导致互相关，这与您的 correl 值不同（不除以标准偏差）。但是我认为您实际上并不需要通过标准差进行标准化才能获得良好的模板匹配。互相关本身非常有效。结果中最大值的位置可以转换为模板的位移w.r.t.

Yes, you will likely gain from using the FFT. Simply pad im_float_2 to the size of im_floatBig. Multiplying in the Fourier domain after taking the complex conjugate of one of the transforms leads to the cross-correlation, which is not the same as your correl value (there is no division by the standard deviations happening). But I don't think you actually need to normalize by the standard deviations to get a good template matching. The cross-correlation works really well by itself. The location of maximum in the result can be translated to a displacement of the template w.r.t. the image.

通过FFT进行互相关的步骤是：

The steps for cross-correlation through the FFT are:

1. 将模板（浮动图片）填充到另一个图片的大小（带有零）。


2. 计算两者的FFT。


3. 翻转结果之一的虚部的符号（复共轭）。


4. 将二者相乘。


5. 计算结果的IFFT。
li>

6. 查找具有最大值的像素的位置。

1. Pad the template (floating image) to the size of the other image (with zeros).
2. Compute the FFT of both.
3. Flip the sign of the imaginary component of one of the results (complex conjugate).
4. Multiply the two.
5. Compute the IFFT of the result.
6. Find the location of the pixel with the largest value.

此像素的位置指示像素填充模板wrt的翻译另一个图像。如果它们最匹配而不进行平移，则最大像素将为（x，y）=（0,0）。如果它是（1,0），则表示沿x偏移了一个像素。方向是取决于您计算复数共轭的方向。请注意，此结果是周期性的，在相反方向上的一个像素移位由位于图像右边缘的最大像素表示。只需做一点试验，以确定如何将位置转换为模板的移位。

The location of this pixel indicates the translation of the padded template w.r.t. the other image. If they best match without translation, the max pixel will be at (x,y)=(0,0). If it is at (1,0) it indicates a one-pixel shift along x. What the direction is depends on which of the two you computed the complex conjugate for. Note that this result is periodic, a one-pixel shift in the opposite direction is indicated by the max pixel being on the right edge of the image. Simply experiment a bit to determine how to translate the location to a shift of your template.

关于您的代码：

1. meanStdDev（im_float_2，im2_Mean，im2_Std）; 是在循环中计算的，即使 im_float_2 不变。

1. meanStdDev(im_float_2, im2_Mean, im2_Std); is computed in the loop, even though im_float_2 doesn't change.

但是您可以不进行任何归一化处理，因为您只是在寻找最大值相关性，并且将搜索中的所有值除以相同的数字不会改变哪个值最大。除以 n_pixels 。

But you could get away with not normalizing by it anyway, since you're just looking for the maximum correlation, and dividing all values in your search by the same number doesn't change which one is the largest. The same applies to the division by n_pixels.

移动 image_window.convertTo （imagePart，CV_8UC1）。在最终找到实际最大值之前，您可能会多次更新当前最大值。如果最终只使用最后一个子窗口，那么将这么多子窗口转换为 CV_U8 没有任何意义。在循环内，您可以更新最大值的（x，y）坐标。

Move image_window.convertTo(imagePart, CV_8UC1) outside the loop. It is likely that you update your current max many times before you finally find the actual max. There is no point in converting so many sub-windows to CV_U8, if you only end up using the last one. Inside the loop you update the (x,y) coordinates of the max. Cast the final location only.

您可能不需要在整个图像中搜索模板。

You probably don't need to search the whole image for your template. It is likely that the object moves only a relatively small amount. You should look only in a small region around the previous known location. This concept is applicable to the FFT method as well: crop out a region of your big image, and pad your template to that size. A smaller FFT is cheaper to compute.

OpenCV按行存储图像。将循环放在行上作为内部循环来优化缓存的使用。

OpenCV stores images row-wise. Put the loop over the rows as the inner loop to optimize your cache usage.

这篇关于解决低FPS的相关代码以计算图像中的偏移的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！