Autograd on sampled locations on feature maps


I have a collection of feature maps, e.g., 4D tensor, and a list of 2D coordinates. I want to extract feature vectors at these coordinates, and use them as input to the next layer. In order not to break autograd, how should I implement it?

My naive idea is to write a for loop and use indexing to extract 2D tensors at each location, but I don’t know whether it will break autograd. Is there any existing function that can performs the similar task?


Just perform the indexing as you would normally do, and autograd should work just fine.
Here is an example of a ROIPooling layer using only autograd operations


I got assert error.
AssertionError: assert(rois.size(1) == 5)

rois from rpn has size(1)=4