如何检查特征描述和匹配是否正确?

huangapple go评论62阅读模式
英文:

How to check if feature descriptors and matches are correct?

问题

我试图找到两个图像之间的共同重叠部分,为此我使用了ORB特征检测器和BEBLID特征描述符。使用这些特征,找到它们之间的单应性矩阵并对齐图像。以下是函数代码:

for pair in image_pairs:
    img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
    img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)

    detector = cv2.ORB_create(1000)
    kpts1 = detector.detect(img1, None)
    kpts2 = detector.detect(img2, None)

    descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
    kpts1, desc1 = descriptor.compute(img1, kpts1)
    kpts2, desc2 = descriptor.compute(img2, kpts2)

    method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
    matcher = cv2.DescriptorMatcher_create(method)
    matches = matcher.match(desc1, desc2, None)

    matches = sorted(matches, key=lambda x: x.distance)

    percentage = 0.2
    keep = int(len(matches) * percentage)
    matches = matches[:keep]
    matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
    cv2.imwrite("feature_match.png", matchedVis)

    ptsA = np.zeros((len(matches), 2), dtype="float")
    ptsB = np.zeros((len(matches), 2), dtype="float")

    for (i, m) in enumerate(matches):
        ptsA[i] = kpts1[m.queryIdx].pt
        ptsB[i] = kpts2[m.trainIdx].pt

    (H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
    (h, w) = img2.shape[:2]
    aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
    cv2.imwrite("wrap.png", aligned)

两个图像成功对齐的样子如下:
如何检查特征描述和匹配是否正确?

而两个图像未成功对齐的样子如下:
如何检查特征描述和匹配是否正确?

image_pairs列表中的一些图像没有共同的重叠部分,因此对齐失败。有没有一种方法可以检测这种失败,甚至在不明确查看warp.png图像的情况下检测成功的对齐?

英文:

I'm trying to find common over laps between two images and for this I am using a ORB feature detector and BEBLID feature descriptor. Using these features, find the homography between them and align the images. The function code is as follows:

for pair in image_pairs:
    img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
    img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)

    detector = cv2.ORB_create(1000)
    kpts1 = detector.detect(img1, None)
    kpts2 = detector.detect(img2, None)

    descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
    kpts1, desc1 = descriptor.compute(img1, kpts1)
    kpts2, desc2 = descriptor.compute(img2, kpts2)

    method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
    matcher = cv2.DescriptorMatcher_create(method)
    matches = matcher.match(desc1, desc2, None)

    matches = sorted(matches, key=lambda x: x.distance)

    percentage = 0.2
    keep = int(len(matches) * percentage)
    matches = matches[:keep]
    matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
    cv2.imwrite("feature_match.png", matchedVis)

    ptsA = np.zeros((len(matches), 2), dtype="float")
    ptsB = np.zeros((len(matches), 2), dtype="float")

    for (i, m) in enumerate(matches):
        ptsA[i] = kpts1[m.queryIdx].pt
        ptsB[i] = kpts2[m.trainIdx].pt

    (H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
    (h, w) = img2.shape[:2]
    aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
    cv2.imwrite("wrap.png", aligned)

A successful alignment of two images looks like:
如何检查特征描述和匹配是否正确?

And an unsuccessful alignment of two images looks like:
如何检查特征描述和匹配是否正确?

Some of the images in image_pairs list have no common overlaps and hence the alignment fails. Is there a way to detect such failures or even detect a successful alignment without explicitly looking at the warp.png images?

答案1

得分: 2

这个线程及其前两个答案对你正在进行的工作是一个有用的资源:

其中所选答案建议的一项操作是检查单应性矩阵的行列式。负行列式表示“翻转图像”,而非常大或非常小的行列式表示放大或缩小的结果。

我认为这是过滤大多数随机抽样一致性算法(RANSAC)可能提供的垃圾单应性的一个很好的方法。

然而,在你提到的“扭曲”图像的示例中,我遗憾地找不到一个合适的数学术语来描述这种变换。

我个人曾经处理过一个更简单的问题,即我有每个我想要变换的图像的四个控制点,但我手动从头实现了“找单应性”的函数。问题出现在当我误匹配控制点时,如下所示:

如何检查特征描述和匹配是否正确?
(请原谅我的可怕的手绘示意图)

其中两个控制点被“交换”,变换后的图像被“扭曲”。对于这个示意图,我实现了一个朴素的、蛮力检查来验证变换后的图像中没有两个边相交。

我在 这个韩文博客中找到了描述这个问题的文章,他们似乎也提供了数学检查任何“异常转换”的方法。根据该博客,检查负的行列式足以捕获这些扭曲的变换。他们还提供了用于确保图像不会过度放大或过度缩小的其他检查。以下是Google翻译的摘录:

如何检查特征描述和匹配是否正确?

请注意,sxsyP 的阈值可以根据你的用例进行调整。

最后,该博客提到这些检查不会捕获“凹变换”(请参阅博客中的图像),你应该手动检查它们。从博客的Google翻译文本:

但在上述 D<0 的条件下不会检查凹的情况。要找到凹的情况,首先将四个点 (0, 0),(1, 0),(0, 1),(1, 0) 变换为一个 2x2 子矩阵,然后检查变换后的结果是否是凹的。要确定它是否是凹的,请参考 [数学] - 计算多边形图形的面积

英文:

This thread and its two top answers are a useful resource for what you are doing:
Detecting garbage homographies from findHomography in OpenCV?

One of the things the selected answer suggests is to check the determinant of the homography matrix. Where negative determinants signals a "flipped image", while a very large or very small determinant signals a blown-up and shrunken results respectively.

I think this is a great way to filter out most of the garbage homography RANSAC happens to give you.
Great supplementary material on determinants of linear transformations


However, in your example case with the image "twisting", I frustratingly cannot find a proper math term for describing the transformation.

I've personally dealt with a simpler problem, where I had 4 control points at the corner of every image I want to transform, but I manually implemented the "find homography" function from scratch. The issue arises when I mismatched the control points like this:

如何检查特征描述和匹配是否正确?
(Pardon my terrible hand-drawn illustration)

Where two of the control points are "swapped" and the transformed image is "twisted". With this illustration, I implemented a naive, brute-force check to verify that no two borders in the transformed image can intersect each other.


I found this Korean blog describing this issue, where they also appear to provide mathematical checks for any "abnormal conversions". According to the blog, checking for negative determinants is enough to catch these twisting transformations. They also provided additional checks for ensuring the image doesn't blow up in size or gets shrunken too much. Here is a Google translated excerpt:

如何检查特征描述和匹配是否正确?

Note that threshold values of sx, sy and P can be adjusted for your usecase.

Finally, the blog mentions that these checks will not catch "concave transformations" (see image in blog) and that you should manually check for them. Google translated text from blog:

> However, the case of concave is not inspected under the above D<0 condition. To find the concave case, first transform the four points (0, 0), (1, 0), (0, 1), (1, 0) into a 2×2 submatrix and then check whether the transformed result is concave. should be inspected. To determine whether it is concave or not, refer to [Mathematics] - Calculating the Area (Area) of a Polygonal Figure.

huangapple
  • 本文由 发表于 2023年2月23日 23:39:53
  • 转载请务必保留本文链接:https://go.coder-hub.com/75547065.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定