opencv通过拉伸的关键点对齐两个图像 [英] opencv align two images by keypoints with stretching

问题描述

我有两张图片

图像 1(对象):

]

image2 是一张白色图片 (500x500)

在 image1 和 image2 中，我已经标记了关键点.我想通过关键点在 image2 上对齐 image1.所以两个关键点与拉伸、缩放和变换image2重叠的目标.

这是我的关键点(csv 文件).image1 中的 image1 和 image2 中的 image2 的坐标是 x 和 y.

object1_x,object1_y,image_x,image_y0,0,80,137286,0,409,42286,198,416,390174,198,331,384158,116,291,1190,97,111,311

我怎样才能用 opencv 和 python 做到这一点?所以结果图应该是这样的(没有红点，红点只是为了展示关键点):

解决方案

概念

从第一组关键点中提取 3 个索引的集合，当从两组关键点进行索引时，这些索引将形成三角形.有了索引，我们可以从两组关键点中获得相应的三角形，从而使我们能够逐个构建扭曲的图像三角形(参见

image2.png (加点):

结果(加分)

代码

导入 cv2将 numpy 导入为 np定义三角形(点):点数 = np.where(点数，点数，1)subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0)))subdiv.insert(列表(点))对于 subdiv.getTriangleList().reshape(-1, 3, 2) 中的点:yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts]定义裁剪(img，pts):x, y, w, h = cv2.boundingRect(pts)img_cropped = img[y: y + h, x: x + w]pts[:, 0] -= xpts[:, 1] -= y返回 img_cropped, pts定义扭曲(img1，img2，pts1，pts2):对于三角形(pts1)中的索引:img1_cropped,triangle1 =crop(img1,pts1[indices])img2_cropped,triangle2 =crop(img2,pts2[indices])变换 = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2))img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101)掩码 = np.zeros_like(img2_cropped)cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0)img2_cropped *= 1 - 掩码img2_cropped += img2_warped * 蒙版img1 = cv2.imread(image1.png")img2 = cv2.imread(image2.png")pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]])pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]])扭曲(img1，img2，pts1，pts2)对于 pts2 中的 pt:cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1)cv2.imshow(原始", img1)cv2.imshow(转换"，img2)cv2.waitKey(0)cv2.destroyAllWindows()

输出

说明

1. 导入必要的库:

导入 cv2将 numpy 导入为 np

2. 定义一个函数，triangles，它将接受坐标数组，points，并为将覆盖该区域的三角形生成数组的 3 个索引列表原始坐标数组的:

定义三角形(点):点数 = np.where(点数，点数，1)subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0)))subdiv.insert(列表(点))对于 subdiv.getTriangleList().reshape(-1, 3, 2) 中的点:产量 [np.where(np.all(points == pt, 1))[0][0] for pt in pts]

3. 定义一个函数crop，它将接收一个图像数组img 和一个包含三个坐标的数组pts.它将返回图像的一个矩形段，刚好适合由三个点组成的三角形，并返回转移到图像左上角的三个坐标数组:

def 裁剪(img，pts):x, y, w, h = cv2.boundingRect(pts)img_cropped = img[y: y + h, x: x + w]pts[:, 0] -= xpts[:, 1] -= y返回 img_cropped, pts

4. 定义一个函数，warp，它将接收 2 个图像数组，img1 和 img2，以及 2 个坐标数组，pts1 和 pts2.它将利用在迭代第一个坐标数组中的三角形之前定义的 triangles 函数，之前定义的 crop 函数在与三角形索引对应的坐标处裁剪两个图像，使用 cv2.warpAffine() 方法在迭代的当前三角形处扭曲图像:

def warp(img1, img2, pts1, pts2):对于三角形(pts1)中的索引:img1_cropped,triangle1 =crop(img1,pts1[indices])img2_cropped,triangle2 =crop(img2,pts2[indices])变换 = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2))img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101)掩码 = np.zeros_like(img2_cropped)cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0)img2_cropped *= 1 - 掩码img2_cropped += img2_warped * 蒙版

5. 阅读您的图片.在您的情况下， img1 是我们想要变形的图像，而 img2 是空白的 500 x 500 图像.另外，定义 2 个坐标数组作为图像的关键点:

img1 = cv2.imread("image1.png")img2 = cv2.imread(image2.png")pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]])pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]])

6. 最后，使用之前定义的warp函数对img1进行warp，使其关键点与img2的关键点重叠并显示结果图像.我将第二个坐标数组中的点绘制到生成的扭曲图像上，以使扭曲过程更易于可视化:

warp(img1, img2, pts1, pts2)对于 pts2 中的 pt:cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1)cv2.imshow(原始", img1)cv2.imshow(转换"，img2)cv2.waitKey(0)cv2.destroyAllWindows()

I have two images

image1 (object):

]3

Original image without marked keypoints:

image2 is a white picture (500x500)

In image1 and in image2 I have marked keypoints. I want to align image1 on image2 by keypoints. So the goal that both keypoints overlaps with stretching, scaling and transforming image2.

This are my keypoints (csv file). The coordinates are x and y for image1 in image1 and for image2 in image2.

object1_x,object1_y,image_x,image_y
0,0,80,137
286,0,409,42
286,198,416,390
174,198,331,384
158,116,291,119
0,97,111,311

How can I do this with opencv and python? So the result image should looks like this (without the red dots, the red dots are only for demonstration the keypoints):

解决方案

The Concept

Extract sets of 3 indices from the first set of keypoints that will form triangles when indexed from both sets of keypoints. With the indices we can get corresponding triangles from both sets of keypoints, allowing us to build the warped image triangle by triangle (see Warp one triangle to another using OpenCV for more details):

image1.png (with added points):

image2.png (with added points):

Result (with added points)

The Code

import cv2
import numpy as np

def triangles(points):
    points = np.where(points, points, 1)
    subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0)))
    subdiv.insert(list(points))
    for pts in subdiv.getTriangleList().reshape(-1, 3, 2):
        yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts]

def crop(img, pts):
    x, y, w, h = cv2.boundingRect(pts)
    img_cropped = img[y: y + h, x: x + w]
    pts[:, 0] -= x
    pts[:, 1] -= y
    return img_cropped, pts

def warp(img1, img2, pts1, pts2): 
    for indices in triangles(pts1):
        img1_cropped, triangle1 = crop(img1, pts1[indices])
        img2_cropped, triangle2 = crop(img2, pts2[indices])
        transform = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2))
        img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101)
        mask = np.zeros_like(img2_cropped)
        cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0)
        img2_cropped *= 1 - mask
        img2_cropped += img2_warped * mask

img1 = cv2.imread("image1.png")
img2 = cv2.imread("image2.png")

pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]])
pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]])

warp(img1, img2, pts1, pts2)

for pt in pts2:
    cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1)

cv2.imshow("Original", img1)
cv2.imshow("Transformed", img2)
cv2.waitKey(0)
cv2.destroyAllWindows()

The Output

The Explanation

1. Import the necessary libraries:

import cv2
import numpy as np

2. Define a function, triangles, that will take in an array of coordinates, points, and yield lists of 3 indices of the array for triangles that will cover the area of the original array of coordinates:

def triangles(points):
    points = np.where(points, points, 1)
    subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0)))
    subdiv.insert(list(points))
    for pts in subdiv.getTriangleList().reshape(-1, 3, 2):
        yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts]

3. Define a function, crop, that will take in an image array, img, and an array of three coordinates, pts. It will return a rectangular segment of the image just large enough to fit the triangle formed by the three point, and return the array of three coordinates transferred to the top-left corner of image:

def crop(img, pts):
    x, y, w, h = cv2.boundingRect(pts)
    img_cropped = img[y: y + h, x: x + w]
    pts[:, 0] -= x
    pts[:, 1] -= y
    return img_cropped, pts

4. Define a function, warp, that will take in 2 image arrays, img1 and img2, and 2 arrays of coordinates, pts1 and pts2. It will utilize the triangles function defined before iterate through the triangles from the first array of coordinates, the crop function defined before to crop both images at coordinates corresponding to the triangle indices and use the cv2.warpAffine() method to warp the image at the current triangle of the iterations:

def warp(img1, img2, pts1, pts2): 
    for indices in triangles(pts1):
        img1_cropped, triangle1 = crop(img1, pts1[indices])
        img2_cropped, triangle2 = crop(img2, pts2[indices])
        transform = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2))
        img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101)
        mask = np.zeros_like(img2_cropped)
        cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0)
        img2_cropped *= 1 - mask
        img2_cropped += img2_warped * mask

5. Read in your images. In your case, img1 is the image we want to warp, and img2 is the blank 500 x 500 image. Also, define 2 array of coordinates to be the keypoints of the images:

img1 = cv2.imread("image1.png")
img2 = cv2.imread("image2.png")

pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]])
pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]])

6. Finally, use the warp function defined before to warp img1 to have its keypoints overlap with the kewpoints of img2 and show the resulting image. I drew the points from the second array of coordinates onto the resulting warped image to make the warping process easier to visualize:

warp(img1, img2, pts1, pts2)

for pt in pts2:
    cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1)

cv2.imshow("Original", img1)
cv2.imshow("Transformed", img2)
cv2.waitKey(0)
cv2.destroyAllWindows()

这篇关于opencv通过拉伸的关键点对齐两个图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！