[Solved]How to do the interpolating of optical flow?

Alex_Hex · July 19, 2017, 6:28am

I used tensorflow before. I just learn pytorch a few days ago.
I want to use pytorch to do something I did with tensorflow before. But I meet a problem, I do not know how to do the interpolating of optical flow.

That is mean, the optical flow has 2 channels such as (Batch_size, 2, height, width). One channel for delta x, another for delta y, using the base grid to add the flow and then sample from origin image.

By the way, in the 2017 CVPR, there is also a paper mention it. Deep feature flow.

Maybe I just start to learn pytorch, it is a bit difficult for me to do interpolating with pytorch. Can I directly use for loop to handle each pixel? Or If you would like to help me, can you show me some demos or examples.

Thanks a lot.

chsasank · July 19, 2017, 8:30am

Are you looking for Upsample?

Alex_Hex · July 19, 2017, 8:44am

Yes. I find what I want: torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’).
You help me a lot, thanks!

Alex_Hex · July 19, 2017, 10:42am

…I find this API torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’). , but not in pytorch version 0.1.12…

chsasank · July 20, 2017, 6:00am

That’s probably because it’s in master branch but not released yet. You can build pytorch from source to get the master version. Read Readme.md in pytorch/pytorch repo for instructions on how to install from source.

Alex_Hex · July 20, 2017, 6:38am

Ok. I solve it. Thanks!

zhuyi490 · November 16, 2017, 7:28pm

Hi Alex,

I have the same question here, I want to use optical flow to warp images. Should I directly use torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’) and nothing else?

I saw some people said that the grid(which in my case, is the optical flow) should be in the range of [-1,1]. Is that true? Because the range of my optical flow is like [-20,20]. So should I divide the flow by 20 before I feed it to the grid_sample function? And is there anything else to notice about? Thank you very much.

lxtGH · March 3, 2018, 8:45am

Hi zhuyi490
I have the same question here, do you know how to do it now ?

IanYeung · June 29, 2018, 3:01am

Hi zhuyi490,

Have you solved this problem now?

sniklaus · July 23, 2018, 11:34pm

You could have a look at the following.

github.com

sniklaus/pytorch-spynet/blob/master/run.py#L135


			def forward(self, tensorInput, tensorFlow):
				if hasattr(self, 'tensorGrid') == False or self.tensorGrid.size(0) != tensorFlow.size(0) or self.tensorGrid.size(2) != tensorFlow.size(2) or self.tensorGrid.size(3) != tensorFlow.size(3):
					torchHorizontal = torch.linspace(-1.0, 1.0, tensorFlow.size(3)).view(1, 1, 1, tensorFlow.size(3)).expand(tensorFlow.size(0), 1, tensorFlow.size(2), tensorFlow.size(3))
					torchVertical = torch.linspace(-1.0, 1.0, tensorFlow.size(2)).view(1, 1, tensorFlow.size(2), 1).expand(tensorFlow.size(0), 1, tensorFlow.size(2), tensorFlow.size(3))


					self.tensorGrid = torch.cat([ torchHorizontal, torchVertical ], 1).cuda()
				# end


				tensorFlow = torch.cat([ tensorFlow[:, 0:1, :, :] / ((tensorInput.size(3) - 1.0) / 2.0), tensorFlow[:, 1:2, :, :] / ((tensorInput.size(2) - 1.0) / 2.0) ], 1)


				return torch.nn.functional.grid_sample(input=tensorInput, grid=(self.tensorGrid + tensorFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='border')
			# end
		# end


		self.modulePreprocess = Preprocess()


		self.moduleBasic = torch.nn.ModuleList([ Basic(intLevel) for intLevel in range(6) ])


		self.moduleBackward = Backward()
	# end

lxtGH · August 11, 2018, 5:58am

Thanks for your code, I want to make sure one thing.
if I want to warp image1(source) to image2(target), the input optical flow is from image2 to image1 rather than image1 to image2 ?? Why?

Qing_En · August 20, 2018, 2:26pm

hi @lxtGH. If I want to warp image1 to image2, the input optical flow is computed from image2 to image1. And should I input this optical flow and image1 together to get image2?

lxtGH · August 23, 2018, 8:25am

Yes, according to the previous code, the source is from image2

github.com

NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py#L139


    self.dc_conv6 = conv(64,       32,  kernel_size=3, stride=1, padding=1,  dilation=1)
    self.dc_conv7 = predict_flow(32)


    for m in self.modules():
        if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
            nn.init.kaiming_normal(m.weight.data, mode='fan_in')
            if m.bias is not None:
                m.bias.data.zero_()




def warp(self, x, flo):
    """
    warp an image/tensor (im2) back to im1, according to the optical flow


    x: [B, C, H, W] (im2)
    flo: [B, 2, H, W] flow


    """
    B, C, H, W = x.size()
    # mesh grid 
    xx = torch.arange(0, W).view(1,-1).repeat(H,1)

983511703 · November 2, 2018, 3:03am

How to do forward warping using img1 and forward flow?

jiahaogeng · November 8, 2018, 1:41pm

Hi, have you solved this problem? If you have, could you share your solution with me?

Benedict_K · November 17, 2018, 8:59pm

Just invert the flow and then warp from img1 to to img2. The flow is a 2d vector field so just multiplying by -1 would invert it. Magnitude remains, just directions is turned by 180 deg.

Arnaud_Roussel · August 5, 2019, 7:59pm

This only works if your flow is uniform. If you use - flow with the forward flow applied on image1 in the grid sampler you will move the wrong pixels. It’s easier to see it in 1D. If you have [0, 0, 1, 0, 0] and a flow of [0, 0, +1, 0, 0] then the warp image is [0, 0, ?? , 1, 0] where ?? being occlusion.

But what the sampler does is to sample from a source given the coordinates in a grid. Your grid with -flow would be:
[0, 1, 2-1 = 1, 3, 4]
After sampling you get:
[0, 0, 0, 0, 0]
because it moved pixel at index 1 instead of pixel at index 2.

That is why the code above is suggesting to use the backward flow instead. In our case that flow would be [0, 0, 0, -1, 0] and then with grid [0, 1, 2, 3-1 = 2, 0]. When using grid sampler from im1 with that grid it will yield:
[0, 0, 1, 1, 0, ]
Which is what we should have without an occlusion mask which should be at index 2.

It sometimes works when you have some general movement in the frame but I prefer to use backward flow instead.

soulslicer · February 29, 2020, 1:37am

Hi all, can i know what is the units that your flow fields are in for the above function. I tried using pixel values directly but I get some extremely warped results

YOGESH_SANKU · December 21, 2021, 6:38am

Hi, I am trying to implement Deep feature flow in PyTorch. Do you have any resources?. I have gone through Scalsol and MM implementation but, if there is any other simple implementation it would be helpful. Thanks in advance.

lizhihao6 · August 8, 2022, 8:35am

You could try this one. ( I am still updating it with bp). GitHub - lizhihao6/Forward-Warp: An optical flow forward warp's lib with backpropagation using pytorch.