I have read the documentations for torch.nn.functional.grid_sample and ended up having questions regarding the “correct” way to backward warp an img.
Suppose I have two images, prev and cur, which are used to calculate the 2D optical flow field(denoted as opt_flow).
The opt_flow denotes the movements of each pixel moving from prev to cur.
Suppose that pixel (3, 5) has optical flow (2, 2), and pixel (5, 7) has optical flow (3, 4).
Then, in order to grid_sample, the grid requires (-2, -2) in (5, 7), and (-3, -4) in (8, 11), but I cannot think of a clear way to create such a grid from the obtained opt_flow, since simply converting to - will place the displacements in wrong positions.
One way will be to apply cur->prev optical flow(inv_opt_flow) to prev instead, which if ideal will have (-2, -2) in (5, 7) and (-3, -4) in (8, 11).
However this seems a bit unintuitive as we are applying the inverse optical flow to the previous image.
Is there a way to create the sampling grid for grid_sample from the forward optical flow?