How to warp the image with optical flow and grid_sample

Hello, guys. I’ve some questions of image warping with optical flows.

I would like to test warp the images by grid_sample, and I’ve got samples(two images with flow) from PWC-Net as follows:
input image from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/data
flow file from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/tmp

def warp(x, flo):
“”"
warp an image/tensor (im2) back to im1, according to the optical flow

x: [B, C, H, W] (im2)
flo: [B, 2, H, W] flow

"""
B, C, H, W = x.size()
# mesh grid
xx = torch.arange(0, W).view(1 ,-1).repeat(H ,1)
yy = torch.arange(0, H).view(-1 ,1).repeat(1 ,W)
xx = xx.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
yy = yy.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
grid = torch.cat((xx ,yy) ,1).float()

if x.is_cuda:
    grid = grid.cuda()
vgrid = Variable(grid) + flo

# scale grid to [-1,1]
vgrid[: ,0 ,: ,:] = 2.0 *vgrid[: ,0 ,: ,:].clone() / max( W -1 ,1 ) -1.0
vgrid[: ,1 ,: ,:] = 2.0 *vgrid[: ,1 ,: ,:].clone() / max( H -1 ,1 ) -1.0

vgrid = vgrid.permute(0 ,2 ,3 ,1)
flo = flo.permute(0 ,2 ,3 ,1)
output = F.grid_sample(x, vgrid)
mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
mask = F.grid_sample(mask, vgrid)

mask[mask <0.9999] = 0
mask[mask >0] = 1

return output*mask

test = cv2.readOpticalFlow(‘reference_frame_0010.flo’)
test = torch.from_numpy(test)
test = test.view(1, 2, H, W)
test = Variable(test).cuda()

test_img2 = cv2.imread(‘frame_0011.png’)
test_img2 = torch.from_numpy(test_img2)
test_img2 = test_img2.view(1, 3, H, W)
test_img2 = Variable(test_img2).cuda()

out = warp(test_img2, test)
out = out.view(H, W, 3)
out = out.cpu()
out = np.float32(out)
cv2.imwrite(‘test3.png’, out*255)

I’ve run the code above and got a result image. As you can see the last image, it is not I expected.
It seems like several part of the images with grid.
Is it right to get a warped target image with optical flow?
If not, how can I get whole image looks like the input color images?

Please, help me to solve this problem.

1 Like

This image is one of the input.

You would have to call transpose on the numpy array or .permute on the tensor to change the dimension order.
The view operation (test.view(1, 2, H, W)) will interleave the pixel values and thus create the wrong image.

3 Likes

I’ve changed all view operations to .permute. It works very well.
Thanks a lot! : )

1 Like

I executed the warp funcion to get back img1 as you post, but the warped image quality was not so good.

. Pay attention to the weapon held by the girl and the shoulder part of the man. Is it because the optical flow was not estimated well?

Hey, did you solve this issue? I think I met the same problem as yours

Hey @LiUzHiAn , did you solve this issue? I think I met the same problem as yours

@Yiran_Xu Not yet.Have you tried different optical flows?

Yep, I tried different flows and found this issue happened to all of them