Python，OpenCV - 一个接一个地对齐和叠加多个图像 [英] Python, OpenCV -- Aligning and overlaying multiple images, one after another

问题描述

我的项目是对齐航拍照片，制作马赛克地图。我的计划是从两张照片开始，将第二张与第一张对齐，然后从两张对齐的图像中创建一个初始马赛克。完成后，我将第三张照片与初始马赛克对齐，然后将第四张照片与其结果对齐，从而逐步构建地图。

My project is to align aerial photos to make a mosaic-map out of them. My plan is to start with two photos, align the second with the first, and create an "initial mosaic" out of the two aligned images. Once that is done, I then align the third photo with the initial mosaic, and then align the fourth photo with the result of that, etc, thereby progressively constructing the map.

我有两种技术可以做到这一点，但更准确的一种，它使用 calcOpticalFlowPyrLK（），只适用于双图像阶段，因为两个输入图像必须大小相同。因此，我尝试了一种新的解决方案，但它不太准确，并且每一步都引入了错误，最终产生了荒谬的结果。

I have two techniques for doing this, but the more accurate one, which makes use of calcOpticalFlowPyrLK(), only works for the two-image phase because the two input images must be the same size. Because of that I tried a new solution, but it is less accurate and the error introduced at every step piles up, eventually producing a nonsensical result.

我的问题是两个 - 折叠，但如果你知道答案，你不必回答两者，除非你想。首先，有没有办法使用类似于 calcOpticalFlowPyrLK（）的东西，但有两个不同大小的图像（这包括任何潜在的解决方法）？第二，有没有办法修改检测器/描述符解决方案以使其更准确？

My question is two-fold, but if you know the answer to one, you don't have to answer both, unless you want to. First, is there a way to use something similar to calcOpticalFlowPyrLK() but with two images of different sizes (this includes any potential workarounds)? And second, is there a way to modify the detector/descriptor solution to make it more accurate?

这是仅适用于两个图像的准确版本：

Here's the accurate version that works only for two images:

# load images
base = cv2.imread("images/1.jpg")
curr = cv2.imread("images/2.jpg")

# convert to grayscale
base_gray = cv2.cvtColor(base, cv2.COLOR_BGR2GRAY)

# find the coordinates of good features to track  in base
base_features = cv2.goodFeaturesToTrack(base_gray, 3000, .01, 10)

# find corresponding features in current photo
curr_features = np.array([])
curr_features, pyr_stati, _ = cv2.calcOpticalFlowPyrLK(base, curr, base_features, curr_features, flags=1)

# only add features for which a match was found to the pruned arrays
base_features_pruned = []
curr_features_pruned = []
for index, status in enumerate(pyr_stati):
    if status == 1:
        base_features_pruned.append(base_features[index])
        curr_features_pruned.append(curr_features[index])

# convert lists to numpy arrays so they can be passed to opencv function
bf_final = np.asarray(base_features_pruned)
cf_final = np.asarray(curr_features_pruned)

# find perspective transformation using the arrays of corresponding points
transformation, hom_stati = cv2.findHomography(cf_final, bf_final, method=cv2.RANSAC, ransacReprojThreshold=1)

# transform the images and overlay them to see if they align properly
# not what I do in the actual program, just for use in the example code
# so that you can see how they align, if you decide to run it
height, width = curr.shape[:2]
mod_photo = cv2.warpPerspective(curr, transformation, (width, height))
new_image = cv2.addWeighted(mod_photo, .5, base, .5, 1)

这是适用于多个图像的不准确的图像（直到错误变得太大）：

Here's the inaccurate one that works for multiple images (until the error becomes too great):

# load images
base = cv2.imread("images/1.jpg")
curr = cv2.imread("images/2.jpg")


# convert to grayscale
base_gray = cv2.cvtColor(self.base, cv2.COLOR_BGR2GRAY)

# DIFFERENCES START
curr_gray = cv2.cvtColor(self.curr_photo, cv2.COLOR_BGR2GRAY)

# create detector, get keypoints and descriptors
detector = cv2.ORB_create()
base_keys, base_desc = detector.detectAndCompute(base_gray, None)
curr_keys, curr_desc = detector.detectAndCompute(curr_gray, None)

matcher = cv2.DescriptorMatcher_create("BruteForce-Hamming")

max_dist = 0.0
min_dist = 100.0

for match in matches:
     dist = match.distance
     min_dist = dist if dist < min_dist else min_dist
     max_dist = dist if dist > max_dist else max_dist

good_matches = [match for match in matches if match.distance <= 3 * min_dist ]

base_matches = []
curr_matches = []
for match in good_matches:
    base_matches.append(base_keys[match.queryIdx].pt)
    curr_matches.append(curr_keys[match.trainIdx].pt)

bf_final = np.asarray(base_matches)
cf_final = np.asarray(curr_matches)

# SAME AS BEFORE

# find perspective transformation using the arrays of corresponding points
transformation, hom_stati = cv2.findHomography(cf_final, bf_final, method=cv2.RANSAC, ransacReprojThreshold=1)

# transform the images and overlay them to see if they align properly
# not what I do in the actual program, just for use in the example code
# so that you can see how they align, if you decide to run it
height, width = curr.shape[:2]
mod_photo = cv2.warpPerspective(curr, transformation, (width, height))
new_image = cv2.addWeighted(mod_photo, .5, base, .5, 1)

最后，这里有一些图片我正在使用：

Finally, here are some images that I'm using:

推荐答案

Homographies组成，所以如果你有 img1 和 img2 以及 img2 和 img3 那么这两个单应性的组成给出了 img1 和 img3 。

Homographies compose, so if you have the homographies between img1 and img2 and between img2 and img3 then the composition of those two homographies gives the homography between img1 and img3.

您的尺寸当然不合适，因为您试图将 img3 与包含<$ c的拼接图像相匹配$ c> img1 和 img2 。但你不需要这样做。在每个连续图像之间都有所有单应性之前，不要缝合它们。然后你可以采取两种方式之一;从后面工作或从前面工作。我将用于例如 h31 引用将 img3 扭曲成 img1 。

Your sizes are off of course because you're trying to match img3 to the stitched image containing img1 and img2. But you don't need to do that. Don't stitch them until you have all the homographies between each successive pair of images. Then you can proceed in one of two ways; work from the back or work from the front. I'll use for e.g. h31 to refer to the homography which warps img3 into coordinates of img1.

从前面（伪代码）：

warp img2 into coordinates of img1 with h21
warp img3 into coordinates of img1 with h31 = h32.dot(h21)
warp img4 into coordinates of img1 with h41 = h43.dot(h31)
...
stitch/blend images together

从后面（伪代码）：

...
warp prev stitched img into coordinates of img3 with h43
stitch warped stitched img with img3
warp prev stitched img into coordinates of img2 with h32
stitch warped stitched img with img2
warp prev stitched img into coordinates of img1 with h21
stitch warped stitched img with img1

这个想法要么是从前面开始，要么将所有东西都扭曲到第一个图像坐标系中，要么从后面开始，扭曲到上一个图像和缝合，然后将拼接的图像扭曲成前一个图像，然后重复。我认为第一种方法可能更容易。在任何一种情况下，你都要担心在单应性估计中误差的传播，因为它们会在多个组合的单应性上积累。

The idea is either you start from the front, and warp everything into the first images coordinate frame, or start from the back, warp to the previous image and stitch, and then warp that stitched image into the previous image, and repeat. I think the first method is probably easier. In either case you have to worry about the propagation of errors in your homography estimation as they will build up over multiple composed homographies.

这是将多个图像与单应性混合在一起的天真方法。更复杂的方法是使用束调整，其考虑所有图像的特征点。然后为了良好的混合，步骤是增益补偿以消除相机增益调整和渐晕，然后多波段混合以防止模糊。请参阅Brown和Lowe的开创性论文此处以及精彩的示例和免费演示软件这里。

This is the naïve approach to blend multiple images together with just the homographies. The more sophisticated method is to use bundle adjustment, which takes into account feature points across all images. Then for good blending the steps are gain compensation to remove camera gain adjustments and vignetting, and then multi-band blending to prevent blurring. See the seminal paper from Brown and Lowe here and a brilliant example and free demo software here.

这篇关于Python，OpenCV - 一个接一个地对齐和叠加多个图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！