Inverse grid sampler

1

image A is the original image, and image B is the image sampled by a sample grid using grid_sample, how to reverse image B to A? the sampling grid is known