在OpenCV中使用StereoBM的差视差图 [英] Bad disparity map using StereoBM in OpenCV

问题描述

我组装了一个立体摄影机，在使用它来产生良好的视差图时遇到了麻烦.这是两个校正后的图像以及我用它们产生的视差图的示例:

I've put together a stereo cam rig and am having trouble using it to produce a good disparity map. Here's an example of two rectified images and the disparity map I produced with them:

如您所见，结果非常糟糕.更改StereoBM的设置不会有太大变化.

As you can see, the results are pretty bad. Changing the StereoBM's settings doesn't change much.

设置

• 两个相机的型号相同，并通过USB连接到我的计算机.
• 它们固定在刚性木板上，因此不会移动.我尽我所能调整了它们的位置，但是当然这并不完美.它们无法移动，因此它们在校准期间和校准之后的位置相同.
• 我使用OpenCV校准了立体声对，并使用OpenCV的 StereoBM 类生成视差图.
• 可能不那么相关，但我使用Python进行编码.

我能想到的问题

我是第一次这样做，所以我还远不​​是专家，但是我猜测问题出在校准或立体校正上，而不是视差图的计算上.我已经尝试了 StereoBM 的所有设置置换，尽管得到了不同的结果，但它们都像上面显示的视差图:黑白补丁.

I'm doing this for the first time, so I'm far from being an expert, but I'm guessing the problem is in the calibration or in the stereo rectification, rather than the computation of the disparity map. I've tried all permutations of settings for the StereoBM and, although I get different results, they're all like the disparity map shown above: Patches of black and white.

据我所知，立体校正应对齐每张图片上的所有点，以便它们通过一条直线(在我的情况下为水平线)相连，这一事实进一步支持了这种想法.如果我将两个校正后的图片彼此相邻地检查，那么显然不是这种情况.右边的图片上的对应点比左边的要高得多.不过，我不确定是校准还是整流.

This idea is further supported by the fact that, as I understand it, stereo rectification should align all points on each picture so that they are connected by a straight (in my case horizontal) line. If I examine both rectified pictures next to each other, it's imediately obvious that this isn't the case. Corresponding points are much higher on the right picture than on the left. I'm not sure whether the calibration or the rectification is the problem, though.

代码

实际的代码包装在对象中-如果您有兴趣完整查看它们，可以在

The actual code is wrapped up in objects - in case you're interested in seeing it in its entirety, it's available on GitHub. Here is a simplified example of what's actually run (of course in the real code I calibrate using more than just 2 pictures):

import cv2
import numpy as np

## Load test images
# TEST_IMAGES is a list of paths to test images
input_l, input_r = [cv2.imread(image, cv2.CV_LOAD_IMAGE_GRAYSCALE)
                    for image in TEST_IMAGES]
image_size = input_l.shape[:2]

## Retrieve chessboard corners
# CHESSBOARD_ROWS and CHESSBOARD_COLUMNS are the number of inside rows and
# columns in the chessboard used for calibration
pattern_size = CHESSBOARD_ROWS, CHESSBOARD_COLUMNS
object_points = np.zeros((np.prod(pattern_size), 3), np.float32)
object_points[:, :2] = np.indices(pattern_size).T.reshape(-1, 2)
# SQUARE_SIZE is the size of the chessboard squares in cm
object_points *= SQUARE_SIZE
image_points = {}
ret, corners_l = cv2.findChessboardCorners(input_l, pattern_size, True)
cv2.cornerSubPix(input_l, corners_l,
                 (11, 11), (-1, -1),
                 (cv2.TERM_CRITERIA_MAX_ITER + cv2.TERM_CRITERIA_EPS,
                  30, 0.01))
image_points["left"] = corners_l.reshape(-1, 2)
ret, corners_r = cv2.findChessboardCorners(input_r, pattern_size, True)
cv2.cornerSubPix(input_r, corners_r,
                 (11, 11), (-1, -1),
                 (cv2.TERM_CRITERIA_MAX_ITER + cv2.TERM_CRITERIA_EPS,
                  30, 0.01))
image_points["right"] = corners_r.reshape(-1, 2)

## Calibrate cameras
(cam_mats, dist_coefs, rect_trans, proj_mats, valid_boxes,
 undistortion_maps, rectification_maps) = {}, {}, {}, {}, {}, {}, {}
criteria = (cv2.TERM_CRITERIA_MAX_ITER + cv2.TERM_CRITERIA_EPS,
            100, 1e-5)
flags = (cv2.CALIB_FIX_ASPECT_RATIO + cv2.CALIB_ZERO_TANGENT_DIST +
         cv2.CALIB_SAME_FOCAL_LENGTH)
(ret, cam_mats["left"], dist_coefs["left"], cam_mats["right"],
 dist_coefs["right"], rot_mat, trans_vec, e_mat,
 f_mat) = cv2.stereoCalibrate(object_points,
                              image_points["left"], image_points["right"],
                              image_size, criteria=criteria, flags=flags)
(rect_trans["left"], rect_trans["right"],
 proj_mats["left"], proj_mats["right"],
 disp_to_depth_mat, valid_boxes["left"],
 valid_boxes["right"]) = cv2.stereoRectify(cam_mats["left"],
                                           dist_coefs["left"],
                                           cam_mats["right"],
                                           dist_coefs["right"],
                                           image_size,
                                           rot_mat, trans_vec, flags=0)
for side in ("left", "right"):
    (undistortion_maps[side],
     rectification_maps[side]) = cv2.initUndistortRectifyMap(cam_mats[side],
                                                           dist_coefs[side],
                                                           rect_trans[side],
                                                           proj_mats[side],
                                                           image_size,
                                                           cv2.CV_32FC1)

## Produce disparity map
rectified_l = cv2.remap(input_l, undistortion_maps["left"],
                        rectification_maps["left"],
                        cv2.INTER_NEAREST)
rectified_r = cv2.remap(input_r, undistortion_maps["right"],
                        rectification_maps["right"],
                        cv2.INTER_NEAREST)
cv2.imshow("left", rectified_l)
cv2.imshow("right", rectified_r)
block_matcher = cv2.StereoBM(cv2.STEREO_BM_BASIC_PRESET, 0, 5)
disp = block_matcher.compute(rectified_l, rectified_r, disptype=cv2.CV_32F)
cv2.imshow("disparity", disp)

这是怎么回事?

推荐答案

事实证明，问题出在可视化上，而不是数据本身.我在某个地方读到 cv2.reprojectImageTo3D 需要一个视差图作为浮点值，这就是为什么我要从 block_matcher.compute 请求 cv2.CV_32F 的原因

It turned out that the problem was the visualization and not the data itself. Somewhere I read that cv2.reprojectImageTo3D required a disparity map as floating point values, which is why I was requesting cv2.CV_32F from block_matcher.compute.

更仔细地阅读OpenCV文档使我认为我在错误地考虑这一点，为了提高速度，我实际上想使用整数而不是浮点数，但是 cv2.imshow的文档尚不清楚16位带符号整数(与16位无符号整数相比)的作用，因此对于可视化，我将值保留为浮点数.

Reading the OpenCV documentation more carefully has led me to think that I was thinking this in error, and I'd actually like to work with integers than floats for the sake of speed, but the documentation for cv2.imshow wasn't clear on what it does with 16 bit signed integers (as compared to 16 bit unsigned), so for the visualization I'm leaving the values as floats.

cv2.imshow 的文档显示32位浮点值假定为介于0和1之间，因此将它们乘以255.255是像素显示为白色时的饱和点.就我而言，这种假设产生了一个二进制图.我手动将其缩放到0-255的范围内，然后将其除以255，以消除OpenCV也执行相同操作的事实.我知道，这是一个可怕的操作，但我这样做只是为了脱机调整 StereoBM ，因此性能并不重要.解决方案如下:

The documentation of cv2.imshow reveals that 32 bit floating point values are assumed to be between 0 and 1, so they're multiplied by 255. 255 is the saturation point at which a pixel is displayed as white. In my case, this assumption produced a binary map. I manually scaled it to the range of 0-255 and then divided it by 255 in order to cancel out the fact that OpenCV does the same as well. I know, it's a horrible operation, but I'm only doing it in order to tune my StereoBM offline so performance is uncritical. The solution looks like this:

# Other code as above
disp = block_matcher.compute(rectified_l, rectified_r, disptype=cv2.CV_32F)
norm_coeff = 255 / disp.max()
cv2.imshow("disparity", disp * norm_coeff / 255)

然后视差图看起来还可以.

Then the disparity map looks okay.

这篇关于在OpenCV中使用StereoBM的差视差图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！