I’ve had this problem in my other thread already, but it isn’t really related, so I moved it to a new thread.
Whenever I try using a custom autograd.Function (i.e., a function that has an explicit backward pass defined), and I combine it with any torch.nn module, the backward pass is never properly executed.
Here’s an MWE containing a simple identity transform as an example:
import torch import torch.autograd import torch.nn # custom autograd function class Identity(torch.autograd.Function): def __init__(self): super(Identity, self).__init__() @staticmethod def forward(self, input): return input.clone() # REASON FOR ERROR: forgot to .clone() here @staticmethod def backward(self, grad_output): print("backward") grad_input = grad_output.clone() return grad_input identity = Identity.apply input = torch.autograd.Variable(torch.randn(1,2,20,20).float(), requires_grad=True) target = torch.autograd.Variable(torch.randn(1,2,20,20).float(), requires_grad=False) # forward pass: # - if any torch.nn function is called before the custom Function, # its backward pass is never executed. intermediate = input # calling backward upon this works fine # intermediate = torch.nn.ReLU()(input) # uncomment this line and it won't work anymore pred = identity(intermediate) loss = torch.nn.MSELoss()(pred,target) loss.backward() # this should always print "backward" to the command line, but it doesn't...
I am running the latest version of PyTorch (Anaconda says 0.2.0, py27hc03bea1_4cu80 [cuda80] soumith), with CUDA 8.0 and Python 2.7.
With some PyTorch versions this seemed to have worked, but now it won’t anymore.
Anybody knows what’s the matter, or else what I’m doing wrong?
Thank you very much for any answers!
EDIT: Silly me. I’ve been trying to solve this for quite some time, and literally 5 minutes after opening this thread I’ve discovered my mistake: I forgot to clone the inputs during the forward pass (code changed above), now it works.
Sorry for the spamming; thread can be closed.