英文:

Image Stitching: Opencv SeamFinder returns patchy masks

问题

I have a setup of 6 cameras (60° horizontal and vertical FoV) with the same center looking outwards with a horizontal rotation of 30 degrees. I am projecting the images onto a sphere and the cameras are located in the center of which, see sketch below.

我的摄像机设置包括6台摄像机（水平和垂直视场角均为60°），它们都朝外以30度的水平旋转。我将图像投影到一个球面上，这些摄像机位于球面的中心，如下图所示：

I am stitching the images together close to this diagram from opencv, but because I know the orientation of the cameras I skip the "registration" part and (for now) omit the blending of the images.

我正在使用OpenCV将图像拼接在一起，与该图示接近，但因为我了解摄像机的方向，所以我跳过了“配准”部分，并（暂时）省略了图像的混合。

My problem is, that the masks returned by the SeamFinder are nonsense. I think it should work, because if I use NoSeamfinder in the simulation the results look good, but I tried every other Seamfinder OpenCV provides and the results are as shown below:

我的问题是，由SeamFinder返回的掩模是无意义的。我认为它应该能够正常工作，因为在模拟中使用NoSeamfinder时结果看起来很好，但我尝试了OpenCV提供的每种其他Seamfinder，结果如下所示：

I have worked with the stitching_detailed.cpp example from opencv to set up the pipeline but I have no idea what might be the issue. I thought that maybe, because I am working within a simulation, the gradients in the images are too sharp to work with, so I applied a Gaussian blur, which did not change anything.

我已经使用OpenCV的stitching_detailed.cpp示例来建立流程，但我不知道问题可能出在哪里。我认为可能是因为我在模拟环境中工作，图像中的梯度太陡峭，所以我应用了高斯模糊，但这并没有改变什么。

I work with cv::Mat in the code and transform them to cv::UMat for the SeamFinder.

我在代码中使用cv::Mat，并将它们转换为cv::UMat以供SeamFinder使用。

<details>
<summary>英文:</summary>

I have a setup of 6 cameras (60&#176; horizontal and vertical FoV) with the same center looking outwards with a horizontal rotation of 30 degrees. I am projecting the images onto a sphere and the cameras are located in the center of which, see sketch below.

[![simplified camera positioning sketch, look from on top of sphere](https://i.stack.imgur.com/pf2Rd.png)](https://i.stack.imgur.com/pf2Rd.png)

I am stitching the images together close to this diagram from opencv, but because I know the orientation of the cameras I skip the &quot;registration&quot; part and (for now) omit the blending of the images.

[![image stitching pipeline from opencv](https://i.stack.imgur.com/V50sM.png)](https://i.stack.imgur.com/V50sM.png)

My problem is, that the masks returned by the SeamFinder are nonsense. I think it should work, because if I use `NoSeamfinder` in the simulation the results look good, but I tried every other Seamfinder OpenCV provides and the results are as shown below:
[![SeamFinder Results](https://i.stack.imgur.com/mWjvQ.jpg)](https://i.stack.imgur.com/mWjvQ.jpg)

I have worked with the stitching_detailed.cpp example from opencv to set up the pipeline but I have no idea what might be the issue. I thought that maybe, because I am working within a simulation, the gradients in the images are too sharp to work with, so I applied a Gaussian blur, which did not change anything.

I have some code snippets below and can post more on demand, but I want to avoid just dumping a whole load here.

I work with `cv::Mat` in the code and transform them to `cv::UMat` for the SeamFinder.

std::vector<cv::Mat> warped_images_mat; // the single images are warped according to the sphere
std::vector<cv::UMat> warped_images_umat;

std::vector<cv::Mat> warped_masks_mat; // Masks of the warped images within the sphere image
std::vector<cv::UMat> warped_masks_umat;

std::vector<cv::Mat> warped_masks_fixed_mat; // These masks are calculated once and used to refresh the masks for the SeamFinder
std::vector<cv::UMat> warped_masks_fixed_umat;

std::vector<cv::Point> corners; // position of the imgs and masks in the sphere image

cv::Ptr<cv::detail::SeamFinder> seam_finder = cv::makePtr<cv::detail::GraphCutSeamFinder>(cv::detail::GraphCutSeamFinderBase::COST_COLOR_GRAD);

// within the ROS2 callback:

this->warped_masks_umat.clear(); // clear the old masks
for(size_t cam_idx = 0; cam_idx < 6; cam_idx++){
int cols = this->warped_images_mat.at(cam_idx).cols;
int rows = this->warped_images_mat.at(cam_idx).rows;

    // std::cout << "rows: " << rows << "; cols: " << cols << std::endl;
    for (int col = 0; col < cols; col++)
        for (int row = 0; row < rows; row++)
            if (this->warped_masks_fixed_mat.at(cam_idx).at<u_char>(row, col) != 0)
                // code to refresh warped image
    this->warped_images_mat.at(cam_idx).getUMat(cv::ACCESS_RW).convertTo(this->warped_images_umat.at(cam_idx), CV_32F);
    this->warped_masks_umat.push_back(this->warped_masks_fixed_umat.at(cam_idx).clone());

}

seam_finder->find(this->warped_images_umat, this->corners, this->warped_masks_umat);

</details>


# 答案1
**得分**: 0

我混淆了OpenCV中的符号。`cv::Point` 中的 `.x` 和 `.y` 被交换了。用于刷新传入图像时它们是这样的，但对于 `SeamFinder` 来说则是相反的。

当我尝试重现收到的评论中的示例时，我注意到了掩膜中的一个独特形状。现在它也可以在完整的掩膜上工作，而不需要裁剪任何内容。

<details>
<summary>英文:</summary>

I mixed up the notations from opencv. The `.x` and `.y` from `cv::Point` were exchanged. It worked like this for refreshing the incoming images but were the wrong way round for the `SeamFinder`.

I noticed this when I tried to reproduce an example from the comment I got and noticed a distinctive shape in the masks. It works now also with the full masks, without cropping anything.

</details>