英文:
I have calculated a flow field in Pytorch. How do I generate a remapped image using this?
问题
我已经在pytorch中使用RAFT模型计算了两帧之间的光流。以下是该代码:
noise_img = noise_img.to(device)
clean_img = clean_img.to(device)
return_index = noise_img.size(1) // 2
aligned_frames = torch.zeros((noise_img.size(0), noise_img.size(1), noise_img.size(2), noise_img.size(3), noise_img.size(4)))
aligned_frames[:, return_index, :, :, :] = noise_img[:, return_index, :, :, :]
for idx in range(noise_img.size(1)):
if not idx == return_index:
curr_frame = noise_img[:, idx, :, :, :]
ref_frame = noise_img[:, return_index, :, :, :]
curr_transf, ref_transf = transforms(curr_frame, ref_frame)
curr_flow = mc_model(curr_transf, ref_transf)[-1] # 获取最终的光流预测
aligned_frames[:, idx, :, :, :] = align_frames(curr_transf, curr_flow)
在上面的代码中,我通过mc_model
(RAFT)传递了两帧以获取光流图。在最后一行,我尝试将当前帧与参考帧对齐。以下是我使用的函数:
def warp_flow(img, flow):
flow_permute = torch.permute(flow, (0, 2, 3, 1))
remapped = torch.nn.functional.grid_sample(img, flow_permute)
return remapped
不幸的是,当将remapped
保存为图像时,它不会返回一个连贯的图像。大多数图像为零,有些看起来像明亮的波纹。我似乎在使用curr_flow
时缺少了一步,但我不太理解是什么问题。
谢谢。
英文:
I have used the RAFT model in pytorch to calculate optical flow between two frames. Here is the code for that:
noise_img = noise_img.to(device)
clean_img = clean_img.to(device)
return_index = noise_img.size(1) // 2
aligned_frames = torch.zeros((noise_img.size(0), noise_img.size(1), noise_img.size(2), noise_img.size(3), noise_img.size(4)))
aligned_frames[:, return_index, :, :, :] = noise_img[:, return_index, :, :, :]
for idx in range(noise_img.size(1)):
if not idx == return_index:
curr_frame = noise_img[:, idx, :, :, :]
ref_frame = noise_img[:, return_index, :, :, :]
curr_transf, ref_transf = transforms(curr_frame, ref_frame)
curr_flow = mc_model(curr_transf, ref_transf)[-1] # Take the final flow prediction
aligned_frames[:, idx, :, :, :] = align_frames(curr_transf, curr_flow)
In the above I am passing two frames through mc_model (RAFT) to return an optical flow map. In the final line I am trying to map the current frame to be aligned with the reference frame. Below is the function I use:
def warp_flow(img, flow):
flow_permute = torch.permute(flow, (0, 2, 3, 1))
remapped = torch.nn.functional.grid_sample(img, flow_permute)
return remapped
Unfortunately, remapped
when saved as an image, does not return a coherent image. Most images are zero with some looking like bright waves. I'm missing a step in using curr_flow
but I don't quite understand what.
Thank you.
答案1
得分: 2
如果我记得正确,RAFT以像素单位输出偏移量,但torch.nn.functional.grid_sample
接受[-1,1]范围内的标准化图像坐标。基本上,您需要使用torch.meshgrid
生成像素坐标,将RAFT生成的流添加到其中,并将其标准化为[-1,1]。这应该作为grid_sample
的输入使用。
英文:
If I remember correctly, RAFT outputs offsets in pixel units, but torch.nn.functional.grid_sample
takes normalized image coordinates in [-1,1]. Basically you need to use torch.meshgrid
to generate pixel coordinates, add the RAFT generated flow to it, and normalize it to [-1,1]. This should be used as the input to grid_sample
.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论