OpenCV立体声匹配/校准 [英] OpenCV Stereo Matching/Calibration

问题描述

我最初是在OpenCV论坛上发布此消息的，但是很遗憾，我没有得到太多的意见/答复，所以我在这里发布是希望有人可能有指导的心得吗?

I'd initially posted this on the OpenCV forums, but unfortunately, I didn't get too many views/replies so I'm posting here with the hopes that someone might have a direction to please suggest?

我正在使用Bumblebee XB3立体相机，它具有3个镜头.我已经花了大约三周的时间阅读有关使用立体声校准和立体声匹配功能的论坛，教程，学习OpenCV书籍以及实际的OpenCV文档.总而言之，我的问题是我生成了一个很好的视差图，但是点云非常差，看起来似乎是歪斜的/压扁的，不能代表实际的场景.

I am using the Bumblebee XB3 Stereo Camera and it has 3 lenses. I've spent about three weeks reading forums, tutorials, the Learning OpenCV book and the actual OpenCV documentation on using the stereo calibration and stereo matching functionality. In summary, my issue is that I have a good disparity map generated but very poor point-clouds, that seem skewed/squished and are not representative of the actual scene.

我到目前为止所做的事情:

What I have done so far:

使用了OpenCV的立体声校准和立体声匹配示例:

Used the OpenCV stereo_calibration and stereo_matching examples to:

使用棋盘图像校准我的立体相机

Calibrated my stereo camera using chess board images

1)原始场景图像: http://answers.opencv.org/upfiles/1380850337241986.jpg
2)使用相机校准后的矩阵校正从相机获得的原始图像
: http://answers.opencv. org/upfiles/13808502665723237.png
3)使用立体匹配(SGBM)
从校正后的图像中生成视差图像: 4)将这些差异投影到3D点云

1) Raw Scene Images: http://answers.opencv.org/upfiles/1380850337241986.jpg
2) Rectified the raw images obtained from the camera using the matrices after camera calibration
: http://answers.opencv.org/upfiles/13808502665723237.png
3) Generated a disparity image from the rectified images using Stereo Matching (SGBM)
: 4) Projected these disparities to a 3D Point Cloud

到目前为止，我为消除问题所做的工作:

What I have done so far as elimination towards my problem:

• 我尝试过第一张和第二张图像，然后是第二张和第三张镜头， 最后是第一和第二.
• 我已经通过更改 距离(更近/更远)
• 我已经使用了20多个立体声对进行校准
• 使用的棋盘大小各不相同:我使用了9x6的棋盘图像 校准，现在改用8x5了
• 我尝试使用块匹配以及SGBM变体并获取
  相对相似的结果.开始
  到目前为止，使用SGBM可获得更好的结果.
• 我改变了视差范围，更改了SAD窗口大小等 没什么改善
  

• I have tried the 1st and 2nd images, then the 2nd and 3rd lenses and finally the 1st and 2nd.
• I've re-run calibration of my chess-board captures by varying the distance (closer/farther away)
• I have used over 20 stereo pairs for the calibration
• Varying Chessboard size used: I have used a 9x6 Chessboard image for calibration and now switched to using a 8x5 one instead
• I've tried using the Block Matching as well as SGBM variants and get
  relatively similar results. Getting
  better results with SGBM so far.
• I've varied the disparity ranges, changed the SAD Window size etc. with little improvement
  

我怀疑问题是:

What I suspect the problem is:

我的视差图像看起来相对可以接受，但是下一步是使用Q矩阵转到3D点云.我怀疑，我没有正确校准相机以生成正确的Q矩阵.不幸的是，我一直在思考如何才能获得更好的Q矩阵.有人可以建议前进的方向吗?

My disparity image looks relatively acceptable, but the next step is to go to 3D Point cloud using the Q matrix. I suspect, I am not calibrating the cameras correctly to generate the right Q matrix. Unfortunately, I've hit the wall in terms of thinking what else I can do to get a better Q matrix. Can someone please suggest ways ahead?

我认为可能有问题的另一件事是我在使用cv :: stereoCalibrate函数时所做的假设.目前，我分别校准每个摄像机以获取摄像机和失真矩阵(cameraMatrix [0]，distCoeffs [0]和cameraMatrix [1]，distCoeffs [1])矩阵，因此它使stereoCalibrate函数的复杂性变得更加容易.

The other thing that I think might be problematic is the assumptions I am making when using the cv::stereoCalibrate function. For the moment, I individually calibrate each camera to get the camera and distortion (cameraMatrix[0], distCoeffs[0] and cameraMatrix[1], distCoeffs[1]) matrices so it makes the complexity for the stereoCalibrate function a little easier.

stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    imageSize, R, T, E, F,
                    TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                    //CV_CALIB_FIX_ASPECT_RATIO +
                    //CV_CALIB_ZERO_TANGENT_DIST +
                    //CV_CALIB_SAME_FOCAL_LENGTH +
                    CV_CALIB_RATIONAL_MODEL 
                    //CV_CALIB_FIX_K3 + CV_CALIB_FIX_K4 + CV_CALIB_FIX_K5
                    );

此外，我认为提及从差异到点云的过程可能会很有用.我正在使用OpenCV的cv :: reprojectImageTo3D，然后将数据写入PCL点云结构.这是相关代码:

Additionally, I think it might be useful to mention how I am going from disparity to point cloud. I am using OpenCV's cv::reprojectImageTo3D and then writing the data to a PCL Point cloud structure. Here is the relevant code:

cv::reprojectImageTo3D( imgDisparity16S, reconstructed3D, Q, false, CV_32F);
  for (int i = 0; i < reconstructed3D.rows; i++)
  {
    for (int j = 0; j < reconstructed3D.cols; j++)
    {
        cv::Point3f cvPoint = reconstructed3D.at<cv::Point3f>(i, j);  
            //Filling in a PCL structure
            pcl::PointXYZRGB point;
            point.x = cvPoint.x;
            point.y = cvPoint.y;
            point.z = cvPoint.z;
            point.rgb = rectified_imgRight.at<cv::Vec3b>(i,j)[0]; //Grey information

            point_cloud_ptr->points.push_back (point);
    }
  }

  point_cloud_ptr->width = (int) point_cloud_ptr->points.size();
  point_cloud_ptr->height = 1;
  pcl::io::savePCDFileASCII("OpenCV-PointCloud.pts", *point_cloud_ptr);

PS:之所以选择上载这些图像是因为场景具有一定的纹理，因此我期望得到一个答复，说场景太均匀.隔断和椅子上的盖子在质地上也很丰富.

PS: The reason I chose to upload these images are that the scene has some texture, so I was anticipating a reply saying the scene is too homogenous. The cover on the partition and the chair as well are quite rich in terms of texture.

几个问题:

Few Questions:

您能帮我删除似乎是点云一部分的图像/视差平面吗?为什么会这样?

Can you help me remove the image/disparity plane that seems to be part of the point cloud? Why is this happening?

我做错了什么明显的东西吗?我会发布我的代码，但是它与提供的OpenCV示例极为相似，并且我认为我没有做任何更有创意的事情.如果可以涉及某个特定部分，我可以.

Is there something obvious I am doing incorrectly? I would post my code, but it is extremely similar to the OpenCV examples provided and I do not think I'm doing anything more creatively. I can if there is a specific section that might be concerning.

在我幼稚的看来，视差图像还可以.但是，从相对不错的视差图像来看，点云绝对不是我所期望的，它会变得更糟.

In my naive opinion, it seems that the disparity image is OK. But the point cloud is definitely nothing I would have expected from a relatively decent disparity image, it is WAY worse.

如果有帮助，我已经提到了相机校准后获得的Q矩阵，以防出现明显的跳跃.将此与学习OpenCV"书进行比较，我认为没有任何明显不正确的地方……

If it helps, I've mentioned the Q matrix I obtain after camera calibration, incase something obvious jumps out. Comparing this to the Learning OpenCV book, I don't think there is anything blatantly incorrect ...

Q: rows: 4
   cols: 4
   data: [ 1., 0., 0., -5.9767076110839844e+002, 0., 1., 0.,
       -5.0785438156127930e+002, 0., 0., 0., 6.8683948509213735e+002, 0.,
       0., -4.4965180874519222e+000, 0. ]

感谢您的阅读，现在我会很感激任何的建议...

Thanks for reading and I'll honestly appreciate any suggestions at this point ...

推荐答案

在使用OpenCV(v.2.4.6)3D重建功能时，我发现了类似的问题. 有些人，例如马丁·佩里斯(Martin Peris)，又重新实施了自己的生活. http://blog.martinperis.com/2012 /01/3d-reconstruction-with-opencv-and-point.html

I found similar problems when using OpenCV (v.2.4.6) 3D reconstruction function. Some people, like Martin Peris, have implemented again by themselves. http://blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html

遗憾的是，有时我的数据在两个实现中都发现了问题. 因此，我认为在这些情况下的问题是由于外部相机参数描述不佳所致.也许，这也是您的情况. :-?

Sadly, sometimes I have found problems in both implementations with my data. So, I supposed my problem in these cases is due to a poor extrinsic camera parameter description. Perhaps, it is your case too. :-?

PS.为了摆脱背景，您需要对其进行细分.或者，至少在重建工作后检查大于阈值的深度值. handleMissingValues标志仅消除无穷大"的点 PS2.请告诉我们您是否解决了这个问题.我认为这对所有社区都是有价值的. thnx

PS. In order to get rid of the background you'll need segment it. Or, at least, check depth values bigger than a threshold once the reconstruction works. handleMissingValues flag only eliminates points at "infinity" PS2. Please, tell us if you solve this problem. I think it is valuable for all comunity. Thnx

这篇关于OpenCV立体声匹配/校准的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！