OpenCV Stitcher类与重叠的固定摄像机 [英] OpenCV Stitcher Class with Overlapping Stationary Cameras

问题描述

我试图使用OpenCV拼接器类来从立体声设置拼接多个帧，其中两个相机都不移动。在多个框架上运行时，我的拼接结果很差。

I'm trying to use the OpenCV stitcher class to stitch multiple frames from a stereo setup, in which neither camera moves. I'm getting poor stitching results when running across multiple frames. I've tried a few different ways, which I'll try to explain here.

使用 stitcher.stitch（）

Using stitcher.stitch( )

给定一对立体视图，我为某些帧运行了以下代码（ VideoFile 是OpenCV VideoCapture 对象的自定义包装：

Given a stereo pair of views, I ran the following code for some frames (VideoFile is a custom wrapper for the OpenCV VideoCapture object):

VideoFile f1( ... );
VideoFile f2( ... );
cv::Mat output_frame;
cv::Stitcher stitcher = cv::Stitcher::createDefault(true);

for( int i = 0; i < num_frames; i++ ) {
    currentFrames.push_back(f1.frame( ));
    currentFrames.push_back(f2.frame( ));
    stitcher.stitch( currentFrames, output_mat );

    // Write output_mat, put it in a named window, etc...

    f1.next_frame();
    f2.next_frame();
    currentFrames.clear();
}

这在每个帧上都有非常好的结果，

This gave really quite good results on each frame, but since the parameters are estimated each frame put in a video you could see the small differences in stitching where the parameters were slightly different.

使用 estimateTransform（）

Using estimateTransform( ) & composePanorama( )

为了解决上述方法的问题，参数只在第一帧，然后使用 composePanorama（）来缝合所有后续帧。

To get past the problem of the above method, I decided to try estimating the parameters only on the first frame, then use composePanorama( ) to stitch all subsequent frames.

for( int i = 0; i < num_frames; i++ ) {
    currentFrames.push_back(f1.frame( ));
    currentFrames.push_back(f2.frame( ));

    if( ! have_transform ) {
        status = stitcher.estimateTransform( currentFrames );
    }

    status = stitcher.composePanorama(currentFrames, output_frame );

    // ... as above
}

似乎存在一个错误（在此处记录），导致两个视图以非常奇怪的方式分开，如图片如下：

Sadly there appears to be a bug (documented here) causing the two views to move apart in a very odd way, as in the images below:

框架1：

Frame 1:

框架2：

Frame 2:

...

框架8：

Frame 8:

显然这是没用的，但我认为这可能是因为该bug在每次调用 composePanorama（）时基本上使内在参数矩阵保持常数。所以我对错误做了一个小补丁，阻止这种情况发生，但然后缝合的结果很差。下面的补丁（ modules / stitching / src / stitcher.cpp ），结果之后：

Clearly this is useless, but I thought it may be just because of the bug, which basically keeps multiplying the intrinsic parameter matrix by a constant each time composePanorama() is called. So I did a minor patch on the bug, stopping this from happening, but then the stitching results were poor. Patch below (modules/stitching/src/stitcher.cpp), results afterwards:

243             for (size_t i = 0; i < imgs_.size(); ++i)
244             {
245                 // Update intrinsics
246                // change following to *=1 to prevent scaling error, but messes up stitching.
247                 cameras_[i].focal *= compose_work_aspect;
248                 cameras_[i].ppx *= compose_work_aspect;
249                 cameras_[i].ppy *= compose_work_aspect; 

结果：

Results:

有没有人有一个线索我如何可以解决这个问题？基本上，我需要做一次转换一次，然后在剩余的帧上使用它（我们说30分钟的视频）。

Does anyone have a clue how I can fix this problem? Basically I need to work out the transformation once, then use it on the remaining frames (we're talking 30mins of video).

我理想地寻找一些关于修补拼图类的建议，但我愿意尝试手工编码一个不同的解决方案。

I'm ideally looking for some advice on patching the stitcher class, but I would be willing to try handcoding a different solution. An earlier attempt which involved finding SURF points, correlating them and finding the homography gave fairly poor results compared to the stitcher class, so I'd rather use it if possible.

推荐答案

所以最后，我黑客的关于stitcher.cpp代码，并得到了一个接近解决方案（但不完美，缝合线仍然移动很多，所以你的里程可能会变化）

So in the end, I hacked about with the stitcher.cpp code and got something close to a solution (but not perfect as the stitching seam still moves about a lot so your mileage may vary).

更改为 stitcher.hpp

Changes to stitcher.hpp

在第136行添加了一个新函数 setCameras（）：

Added a new function setCameras() at line 136:

void setCameras( std::vector<detail::CameraParams> c ) {

     this->cameras_ = c;
 }`

添加了一个新的私有成员变量来跟踪这是否是我们的第一个估计：

Added a new private member variable to keep track of whether this is our first estimation:

bool _not_first;

更改为 stitcher.cpp

Changes to stitcher.cpp

在 estimateTransform（）（行〜100）：

this->not_first = 0;
images.getMatVector(imgs_);
// ... 

在 composePanorama code>（line〜227）：

In composePanorama() (line ~227):

// ...
compose_work_aspect = compose_scale / work_scale_;

// Update warped image scale
if( !this->not_first ) { 
    warped_image_scale_ *= static_cast<float>(compose_work_aspect);
    this->not_first = 1;
}   

w = warper_->create((float)warped_image_scale_);
// ...

代码调用 object：

Code calling stitcher object:

所以基本上，我们创建一个拼接器对象，然后在第一帧外部拼接类）。然后，拼接器将沿着线的某处破坏本征矩阵，导致下一帧乱七八糟。因此，在我们处理它之前，我们只是使用我们从类中提取的那些相机重置。

So basically, we create a stitcher object, then get the transform on the first frame (storing the camera matrices outside the stitcher class). The stitcher will then break the Intrinsic Matrix somewhere along the line causing the next frame to mess up. So before we process it, we just reset the cameras using the ones we extracted from the class.

警告，我必须有一些错误检查，不使用默认设置生成估计 - 您可能需要在获取结果之前使用 setPanoConfidenceThresh（...）迭代地降低置信度阈值。

Be warned, I had to have some error checking in case the stitcher couldn't produce an estimation with the default settings - you may need to iteratively decrease the confidence threshold using setPanoConfidenceThresh(...) before you get a result.

cv::Stitcher stitcher = cv::Stitcher::createDefault(true);
std::vector<cv::detail::CameraParams> cams;
bool have_transform = false;

for( int i = 0; i < num_frames; i++ ) {
        currentFrames.push_back(f1.frame( ));
        currentFrames.push_back(f2.frame( ));

        if( ! have_transform ) {
            status = stitcher.estimateTransform( currentFrames );
            have_transform = true;
            cams = stitcher.cameras();

            // some code to check the status of the stitch and handle errors...
        }

        stitcher.setCameras( cams );
        status = stitcher.composePanorama(currentFrames, output_frame );

        // ... Doing stuff with the panorama
}

请注意，这是一个非常黑客的OpenCV代码，这将使更新到较新的版本痛苦。不幸的是，我缺少时间，所以一个讨厌的黑客是我可以得到的一切！

Please be aware that this is very much a hack of the OpenCV code, which is going to make updating to a newer version a pain. Unfortunately I was short of time so a nasty hack was all I could get round to!

这篇关于OpenCV Stitcher类与重叠的固定摄像机的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！