如何找到两个图像之间的所有共享区域 [英] how to find all shared regions between two images

问题描述

我一直在尝试寻找可以自动找到两幅图像之间所有共享区域的东西，而基于像素匹配或差异来显式地不是，我基本上在公平的情况下一无所获一点搜索.

说我有以下两个图像，在这种情况下，是网站屏幕截图.首先是基准":

和第二个非常相似，但带有一些修改过的CSS，因此整个块都被移动了.文本内容没有变化，盒子尺寸也没有变化，只是重新定位了一些元素:

在这种情况下(但实际上在每两个其他情况下，要比较其中一个是另一个的派生的两个图像)，它们的像素差异实际上对于观察变化是毫无用处的:

实际上，即使我们应用了一些简单的差异夸张，结果仍然是 still 毫无用处，因为我们仍在查看像素差异，而不是根据更改的差异，因此我们不会(以任何方式)查看对视觉信息的实际修改:

所以这就像比较两本书，然后根据 n 的多少个值来确定这些书是否不同，我们可以找到哪些 book1.letters [n]！= book2.letters[n] ...

因此，我正在寻找一种计算相似性区域的方法，该方法显示两个图像的哪些部分编码相同的信息，但不一定在相同的边界框中.

例如，在上面的两个图像中，几乎所有数据都是相同的，只是其中一些部分已重定位.唯一真正的区别是那里有一个神秘的空白.

具有相似区域的颜色编码:

和对应关系:

我找不到单个工具来执行此操作，甚至找不到可以使用opencv或类似技术实现此功能的教程.也许我在寻找错误的术语，也许没有人为此写过一个图像比较工具(这似乎令人难以置信?)，所以冒着偏离主题的风险:我解决方案

回答我自己的问题: opencv ( for python )与 SSIM 比较，捕获相对于bbox轮廓的各种差异到第二张图片

针对第二张图片中的每个轮廓，对第一张图片执行模板匹配，它告诉我们diff轮廓是否为变化".或翻译".

在代码中，假设具有相同尺寸的两个图像 imageA 和 imageB :

  import cv2导入imutils从skimage.metrics导入structure_similarity#...一堆函数将要在这里...diffs = compare(imageA，imageB，gray(imageA)，gray(imageB)，[])如果len(diffs)>0:highlight_diffs(imageA，imageB，diffs)别的:打印(未检测到差异") 

具有:

 <代码> def灰色(img):返回cv2.cvtColor(img，cv2.COLOR_BGR2GRAY)def compare(or1，or2，im1，img2，diffs):(分数，差异)=结构相似度(im1，img2，full = True)diff =(diff * 255).astype("uint8")thresh = cv2.threshold(diff，0，255，cv2.THRESH_BINARY_INV |cv2.THRESH_OTSU)[1]等高线= cv2.findContours(thresh.copy()，cv2.RETR_EXTERNAL，cv2.CHAIN_APPROX_SIMPLE)轮廓= imutils.grab_contours(轮廓)#聚合轮廓，丢弃重复项对于轮廓中的c:(x，y，w，h)= cv2.boundingRect(c)区域= [x，y，x + w，y + h]尝试:diffs.index(区域)除了ValueError:diffs.append(区域)返回差异 

现在，假定 cv2.RETR_EXTERNAL 仅产生外部轮廓"，例如如果在其他差异之间内部有差异(例如，框的边框颜色已更改，并且框内的某些文本也已更改)，则它应仅产生一个框，即外框("external")./p>

除了不是那样，所以我写了一个愚蠢的函数，天真地除杂了内箱:

 <代码> def filter_diffs(diffs):def not_contained(e，diffs):对于差异中的t:如果e [0]>t [0]和e [2]<t [2]和e [1]>t [1]和e [3]<t [3]:返回False返回True返回[如果not_contained(e，diffs)，则表示差异中e的e] 

然后将其用于使用彩色矩形突出显示差异的函数中.

  RED =(0,0,255)def Highlight_diffs(a，b，diffs):差异= b.copy()对于filter_diffs(diffs)中的区域:x1，y1，x2，y2 =面积cv2.rectangle(diffed，(x1，y1)，(x2，y2)，RED，2)cv2.imshow(差异"，差异) 

这使我们成为第一部分.拍摄Stackoverflow的屏幕截图，然后向下移动左侧广告，然后为-yellow-100 CSS变量重新着色，然后再绘制另一个屏幕截图:

这找到五个差异，但其中两个并不是真正的差异".从某种意义上说，这是新内容或已删除的内容，但这是我们将某件商品放下了"的结果.

因此，让我们添加模板匹配项:

  def highlight_diffs(a，b，diffs):差异= b.copy()对于filter_diffs(diffs)中的区域:x1，y1，x2，y2 =面积#这是重定位还是添加/删除?org = find_in_original(a，b，area)如果org不是None:cv2.rectangle(a，(org [0]，org [1])，(org [2]，org [3])，蓝色，2)cv2.rectangle(diffed，(x1，y1)，(x2，y2)，BLUE，2)别的:cv2.rectangle(diffed，(x1 + 2，y1 + 2)，(x2-2，y2-2)，GREEN，1)cv2.rectangle(diffed，(x1，y1)，(x2，y2)，RED，2)cv2.imshow(原始"，a)cv2.imshow(差异"，差异)cv2.waitKey(0) 

使用以下代码进行模板匹配，并为是我们发现的匹配实际上是很好的"设置了非常严格的阈值:

  def find_in_original(a，b，area):作物= b [面积[1]:面积[3]，面积[0]:面积[2]]结果= cv2.matchTemplate(crop，a，cv2.TM_CCOEFF_NORMED)(minVal，maxVal，minLoc，maxLoc)= cv2.minMaxLoc(result)(startX，startY)= maxLocendX = startX +(区域[2]-区域[0])endY = startY +(区域[3]-区域[1])ocrop = a [startY:endY，startX:endX]#这基本上需要接近完美的匹配#让我们将其视为已移动"地区而不是#A和B之间的真正区别.如果structure_similarity(gray(ocrop)，grey(crop))> = 0.99:返回[startX，startY，endX，endY] 

我们现在可以比较原始图像和修改过的图像，并看到广告在修改后的图像中移动了，而不是新内容"，并且我们可以看到在原始位置可以找到它:

就是这样，我们有一个视觉差异，它实际上告诉我们有关更改的一些有用信息，而不是告诉我们哪个像素碰巧是另一种颜色.

我们可以将模板匹配阈值降低一些，例如0.95，在这种情况下，空格框也将最终与原始图像匹配，但是因为它只是空格，所以它将与大多数无意义的东西匹配(在这种情况下，它将与原始图片右下角的空白匹配).

当然，生活质量的改善将是在颜色之间循环，以便各个移动的部件都可以通过它们共享的颜色相互关联，但这是任何人都可以自己在此代码之上添加的东西.

I've been trying to find something that automatically finds all shared regions between two images, explicitly not based on pixel-matching or differencing, and I'm basically coming up with nothing after a fair bit of searching.

Say I have the following two images, in this case, website screenshots. The first the "baseline":

and the second very similar but with some modified CSS so entire blocks have been moved around. No text content changes, no box dimension changes, just some elements repositioned:

In this case (but also in literally every other case where two images where one is a derivative of the other are to be compared) their pixel diff is effectively useless for seeing what changed:

In fact, even if we apply some simple diff exaggeration, the result is still fairly useless because we're still looking at pixel diffs, instead of diffs based on what changed, so we won't (in any way) be looking at the actual modifications to the visual information:

So this is like comparing two books and then deciding the books are different based on how many values for n we can find for which book1.letters[n] != book2.letters[n]...

So, what I'm looking for is a way to compute regions of similarity, showing which parts of the two images encode the same information, but not necessarily in the same bounding boxes.

For instance, in the above two images, almost all the data is the same, just with some parts relocated. The only true difference is that there's mystery whitespace.

With similar regions color coded:

and the correspondence:

I can't find a single tool to do this, and I can't even find tutorials that allow for implementation of this using opencv or similar technologies. Maybe I'm looking for the wrong terms, maybe no one actually ever wrote an image comparison tool for this (which seems beyond belief?), so at the risk of this being off topic: I searched and researched as much as I can, here. What are my options if I need this as a tool that can be run as part of a normal (open source) tool chain for QA/testing? (so: not some expensive plugin to equally expensive commercial software).

解决方案

To answer my own question: opencv (for python) paired with scikit-image can pretty much get us there in two steps.

1. perform an SSIM comparison between the two images, capturing the various differences in bbox contours relative to the second image
2. for each contour in the second image, perform template matching with respect to the first image, which tells us if a diff contour is a "change" or a "translation".

In code, assuming two images imageA and imageB, with the same dimensions:

import cv2
import imutils
from skimage.metrics import structural_similarity

# ...a bunch of functions will be going here...

diffs = compare(imageA, imageB, gray(imageA), gray(imageB), [])

if len(diffs) > 0:
    highlight_diffs(imageA, imageB, diffs)

else:
    print("no differences detected")

with:

def gray(img):
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

def compare(or1, or2, im1, img2, diffs):
    (score, diff) = structural_similarity(im1, img2, full=True)
    diff = (diff * 255).astype("uint8")

    thresh = cv2.threshold(diff, 0, 255,
        cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
    contours = cv2.findContours(thresh.copy(),
        cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    contours = imutils.grab_contours(contours)

    # aggregate the contours, throwing away duplicates
    for c in contours:
        (x, y, w, h) = cv2.boundingRect(c)
        region = [x, y, x + w, y + h]
        try:
            diffs.index(region)
        except ValueError:
            diffs.append(region)

    return diffs

Now, cv2.RETR_EXTERNAL is supposed to only yield "External contours", e.g. if there's diffs inside other diffs (say a box's border color changed, and some text inside the box also changed), it should just yield one box, being the outer ("external") box.

Except that's not what it does, so I wrote a dumb function that naively weeds inner boxes:

def filter_diffs(diffs):
    def not_contained(e, diffs):
        for t in diffs:
            if e[0] > t[0] and e[2] < t[2] and e[1] > t[1] and e[3] < t[3]:
                return False
        return True

    return [e for e in diffs if not_contained(e, diffs)]

which then gets used in the function that highlights the differences using color rectangles.

RED = (0,0,255)

def highlight_diffs(a, b, diffs):
    diffed = b.copy()

    for area in filter_diffs(diffs):
        x1, y1, x2, y2 = area
        cv2.rectangle(diffed, (x1, y1), (x2, y2), RED, 2)

    cv2.imshow("Diffed", diffed)

This gets us the first part. Taking a screenshot of Stackoverflow, and then another screenshot after moving the left advertisement down, and recoloring the --yellow-100 CSS variable:

This finds five diffs, but two of them aren't really "diffs" in the sense that it's new or removed content, but rather it's the result of "we moved a thing down".

So, let's add in the template matching:

def highlight_diffs(a, b, diffs):
    diffed = b.copy()

    for area in filter_diffs(diffs):
        x1, y1, x2, y2 = area

        # is this a relocation, or an addition/deletion?
        org = find_in_original(a, b, area)
        if org is not None:
            cv2.rectangle(a, (org[0], org[1]), (org[2], org[3]), BLUE, 2)
            cv2.rectangle(diffed, (x1, y1), (x2, y2), BLUE, 2)
        else:
            cv2.rectangle(diffed, (x1+2, y1+2), (x2-2, y2-2), GREEN, 1)
            cv2.rectangle(diffed, (x1, y1), (x2, y2), RED, 2)

    cv2.imshow("Original", a)
    cv2.imshow("Diffed", diffed)
    cv2.waitKey(0)

With the following code for the template matching, with an incredibly strict threshold for "is the match we found actually good":

def find_in_original(a, b, area):
    crop = b[area[1]:area[3], area[0]:area[2]]
    result = cv2.matchTemplate(crop, a, cv2.TM_CCOEFF_NORMED)

    (minVal, maxVal, minLoc, maxLoc) = cv2.minMaxLoc(result)
    (startX, startY) = maxLoc
    endX = startX + (area[2] - area[0])
    endY = startY + (area[3] - area[1])
    ocrop = a[startY:endY, startX:endX]

    # this basically needs to be a near-perfect match
    # for us to consider it a "moved" region rather than
    # a genuine difference between A and B.
    if structural_similarity(gray(ocrop), gray(crop)) >= 0.99:
        return [startX, startY, endX, endY]

We can now compare the original and modified image, and see that the ad got moved in the modified image, rather than being "new content", and we can see where it can be found in the original:

And that's it, we have a visual diff that actually tells us something useful about the changes, rather than telling us which pixel happens to be a different colour.

We could bring down the template matching threshold down a little to, say, 0.95, in which case the whitespace box would also end up matched to the original image, but because it's just whitespace it'll get matched to something mostly meaningless (in this particular case, it'll match it to the whitespace in the lower right of the original).

Of course, quality of life improvements would be to cycle through colours so that various moved parts can all be related to each other by their shared colour, but that's the kind of thing that anyone can probably tack on top of this code themselves.

这篇关于如何找到两个图像之间的所有共享区域的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！