[Solved]How to do the interpolating of optical flow?

I used tensorflow before. I just learn pytorch a few days ago.
I want to use pytorch to do something I did with tensorflow before. But I meet a problem, I do not know how to do the interpolating of optical flow.

That is mean, the optical flow has 2 channels such as (Batch_size, 2, height, width). One channel for delta x, another for delta y, using the base grid to add the flow and then sample from origin image.

By the way, in the 2017 CVPR, there is also a paper mention it. Deep feature flow.

Maybe I just start to learn pytorch, it is a bit difficult for me to do interpolating with pytorch. Can I directly use for loop to handle each pixel? Or If you would like to help me, can you show me some demos or examples.

Thanks a lot.

Are you looking for Upsample?

1 Like

Yes. I find what I want: torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’).
You help me a lot, thanks!

…I find this API torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’). , but not in pytorch version 0.1.12…

That’s probably because it’s in master branch but not released yet. You can build pytorch from source to get the master version. Read Readme.md in pytorch/pytorch repo for instructions on how to install from source.

Ok. I solve it. Thanks!

Hi Alex,

I have the same question here, I want to use optical flow to warp images. Should I directly use torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’) and nothing else?

I saw some people said that the grid(which in my case, is the optical flow) should be in the range of [-1,1]. Is that true? Because the range of my optical flow is like [-20,20]. So should I divide the flow by 20 before I feed it to the grid_sample function? And is there anything else to notice about? Thank you very much.

1 Like

Hi zhuyi490
I have the same question here, do you know how to do it now ?

1 Like

Hi zhuyi490,

Have you solved this problem now?

You could have a look at the following.


Thanks for your code, I want to make sure one thing.
if I want to warp image1(source) to image2(target), the input optical flow is from image2 to image1 rather than image1 to image2 ?? Why?

hi @lxtGH. If I want to warp image1 to image2, the input optical flow is computed from image2 to image1. And should I input this optical flow and image1 together to get image2?

Yes, according to the previous code, the source is from image2

How to do forward warping using img1 and forward flow?

Hi, have you solved this problem? If you have, could you share your solution with me?

Just invert the flow and then warp from img1 to to img2. The flow is a 2d vector field so just multiplying by -1 would invert it. Magnitude remains, just directions is turned by 180 deg.

This only works if your flow is uniform. If you use - flow with the forward flow applied on image1 in the grid sampler you will move the wrong pixels. It’s easier to see it in 1D. If you have [0, 0, 1, 0, 0] and a flow of [0, 0, +1, 0, 0] then the warp image is [0, 0, ?? , 1, 0] where ?? being occlusion.

But what the sampler does is to sample from a source given the coordinates in a grid. Your grid with -flow would be:
[0, 1, 2-1 = 1, 3, 4]
After sampling you get:
[0, 0, 0, 0, 0]
because it moved pixel at index 1 instead of pixel at index 2.

That is why the code above is suggesting to use the backward flow instead. In our case that flow would be [0, 0, 0, -1, 0] and then with grid [0, 1, 2, 3-1 = 2, 0]. When using grid sampler from im1 with that grid it will yield:
[0, 0, 1, 1, 0, ]
Which is what we should have without an occlusion mask which should be at index 2.

It sometimes works when you have some general movement in the frame but I prefer to use backward flow instead.

1 Like

Hi all, can i know what is the units that your flow fields are in for the above function. I tried using pixel values directly but I get some extremely warped results

Hi, I am trying to implement Deep feature flow in PyTorch. Do you have any resources?. I have gone through Scalsol and MM implementation but, if there is any other simple implementation it would be helpful. Thanks in advance.

You could try this one. ( I am still updating it with bp). GitHub - lizhihao6/Forward-Warp: An optical flow forward warp's lib with backpropagation using pytorch.