英文:
How to check if feature descriptors and matches are correct?
问题
我试图找到两个图像之间的共同重叠部分,为此我使用了ORB特征检测器和BEBLID特征描述符。使用这些特征,找到它们之间的单应性矩阵并对齐图像。以下是函数代码:
for pair in image_pairs:
img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)
detector = cv2.ORB_create(1000)
kpts1 = detector.detect(img1, None)
kpts2 = detector.detect(img2, None)
descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
kpts1, desc1 = descriptor.compute(img1, kpts1)
kpts2, desc2 = descriptor.compute(img2, kpts2)
method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
matcher = cv2.DescriptorMatcher_create(method)
matches = matcher.match(desc1, desc2, None)
matches = sorted(matches, key=lambda x: x.distance)
percentage = 0.2
keep = int(len(matches) * percentage)
matches = matches[:keep]
matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
cv2.imwrite("feature_match.png", matchedVis)
ptsA = np.zeros((len(matches), 2), dtype="float")
ptsB = np.zeros((len(matches), 2), dtype="float")
for (i, m) in enumerate(matches):
ptsA[i] = kpts1[m.queryIdx].pt
ptsB[i] = kpts2[m.trainIdx].pt
(H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
(h, w) = img2.shape[:2]
aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
cv2.imwrite("wrap.png", aligned)
两个图像成功对齐的样子如下:
而两个图像未成功对齐的样子如下:
在image_pairs
列表中的一些图像没有共同的重叠部分,因此对齐失败。有没有一种方法可以检测这种失败,甚至在不明确查看warp.png
图像的情况下检测成功的对齐?
英文:
I'm trying to find common over laps between two images and for this I am using a ORB feature detector and BEBLID feature descriptor. Using these features, find the homography between them and align the images. The function code is as follows:
for pair in image_pairs:
img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)
detector = cv2.ORB_create(1000)
kpts1 = detector.detect(img1, None)
kpts2 = detector.detect(img2, None)
descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
kpts1, desc1 = descriptor.compute(img1, kpts1)
kpts2, desc2 = descriptor.compute(img2, kpts2)
method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
matcher = cv2.DescriptorMatcher_create(method)
matches = matcher.match(desc1, desc2, None)
matches = sorted(matches, key=lambda x: x.distance)
percentage = 0.2
keep = int(len(matches) * percentage)
matches = matches[:keep]
matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
cv2.imwrite("feature_match.png", matchedVis)
ptsA = np.zeros((len(matches), 2), dtype="float")
ptsB = np.zeros((len(matches), 2), dtype="float")
for (i, m) in enumerate(matches):
ptsA[i] = kpts1[m.queryIdx].pt
ptsB[i] = kpts2[m.trainIdx].pt
(H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
(h, w) = img2.shape[:2]
aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
cv2.imwrite("wrap.png", aligned)
A successful alignment of two images looks like:
And an unsuccessful alignment of two images looks like:
Some of the images in image_pairs
list have no common overlaps and hence the alignment fails. Is there a way to detect such failures or even detect a successful alignment without explicitly looking at the warp.png
images?
答案1
得分: 2
这个线程及其前两个答案对你正在进行的工作是一个有用的资源:
其中所选答案建议的一项操作是检查单应性矩阵的行列式。负行列式表示“翻转图像”,而非常大或非常小的行列式表示放大或缩小的结果。
我认为这是过滤大多数随机抽样一致性算法(RANSAC)可能提供的垃圾单应性的一个很好的方法。
然而,在你提到的“扭曲”图像的示例中,我遗憾地找不到一个合适的数学术语来描述这种变换。
我个人曾经处理过一个更简单的问题,即我有每个我想要变换的图像的四个控制点,但我手动从头实现了“找单应性”的函数。问题出现在当我误匹配控制点时,如下所示:
其中两个控制点被“交换”,变换后的图像被“扭曲”。对于这个示意图,我实现了一个朴素的、蛮力检查来验证变换后的图像中没有两个边相交。
我在 这个韩文博客中找到了描述这个问题的文章,他们似乎也提供了数学检查任何“异常转换”的方法。根据该博客,检查负的行列式足以捕获这些扭曲的变换。他们还提供了用于确保图像不会过度放大或过度缩小的其他检查。以下是Google翻译的摘录:
请注意,sx
、sy
和 P
的阈值可以根据你的用例进行调整。
最后,该博客提到这些检查不会捕获“凹变换”(请参阅博客中的图像),你应该手动检查它们。从博客的Google翻译文本:
但在上述 D<0 的条件下不会检查凹的情况。要找到凹的情况,首先将四个点 (0, 0),(1, 0),(0, 1),(1, 0) 变换为一个 2x2 子矩阵,然后检查变换后的结果是否是凹的。要确定它是否是凹的,请参考 [数学] - 计算多边形图形的面积。
英文:
This thread and its two top answers are a useful resource for what you are doing:
Detecting garbage homographies from findHomography in OpenCV?
One of the things the selected answer suggests is to check the determinant of the homography matrix. Where negative determinants signals a "flipped image", while a very large or very small determinant signals a blown-up and shrunken results respectively.
I think this is a great way to filter out most of the garbage homography RANSAC happens to give you.
Great supplementary material on determinants of linear transformations
However, in your example case with the image "twisting", I frustratingly cannot find a proper math term for describing the transformation.
I've personally dealt with a simpler problem, where I had 4 control points at the corner of every image I want to transform, but I manually implemented the "find homography" function from scratch. The issue arises when I mismatched the control points like this:
(Pardon my terrible hand-drawn illustration)
Where two of the control points are "swapped" and the transformed image is "twisted". With this illustration, I implemented a naive, brute-force check to verify that no two borders in the transformed image can intersect each other.
I found this Korean blog describing this issue, where they also appear to provide mathematical checks for any "abnormal conversions". According to the blog, checking for negative determinants is enough to catch these twisting transformations. They also provided additional checks for ensuring the image doesn't blow up in size or gets shrunken too much. Here is a Google translated excerpt:
Note that threshold values of sx
, sy
and P
can be adjusted for your usecase.
Finally, the blog mentions that these checks will not catch "concave transformations" (see image in blog) and that you should manually check for them. Google translated text from blog:
> However, the case of concave is not inspected under the above D<0 condition. To find the concave case, first transform the four points (0, 0), (1, 0), (0, 1), (1, 0) into a 2×2 submatrix and then check whether the transformed result is concave. should be inspected. To determine whether it is concave or not, refer to [Mathematics] - Calculating the Area (Area) of a Polygonal Figure.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论