Using gather() in forward function

1

When I use gather in forward,I get this error:

RuntimeError: save_for_backward can only save input or output tensors,
 but argument 0 doesn't satisfy this condition

It seems code below is not correct

        inds=inds.view(N,C,-1).data
        out=torch.gather(target,dim=2,index=inds)

After searching here, I foud this one-----Nn module using index_select to the answer there,I changed the code.

        inds=inds.view(N,C,-1)
        out=torch.gather(target,dim=2,index=inds)

But then I get another eror-------AssertionError: Gather can’t differentiate the index.Could anyone tell me why?Thanks a lot.

2

When working with autograd, you should always use Variables and so never use the .data field. This is why there was an error in save_for_backward: because the indices were not a Variable.
The second error you get is because inds requires gradients (because some elements before in the gradient have learnable parameters and thus require gradients to update), but the gather function cannot provide gradients wrt the indices.
You can call .detach() on the inds Variable to detach it from it’s history such that it does not require gradients anymore. Note that when doing that, no gradients will be backpropagated by this Variable.

2 Likes
3

Thanks,I got it.But if I want to preserve the gradients,what should I do?Is there any way besides gather()?

4

You cannot get the gradients for the indices, they don’t exist.
You get proper gradients for the tensor you’re gathering from though.

2 Likes
5

Thank you .You are right.I needn’t to calculate that.Truly grateful for your help.:+1: