I’m using a resnet18 from torchvision. When running the forward pass, I get the following warning:
UserWarning: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is deprecated and will be forbidden starting version 1.6. You can remove this warning by cloning the output of the custom Function.
And when I run the backward pass, I get
Module backward hook for grad_input is called before the grad_output one. This happens because the gradient in your nn.Module flows to the Module's input without passing through the Module's output. Make sure that the output depends on the input and that the loss is computed based on the output.
The error seems to stem from
result = torch.relu_(input)
This didn’t happen previously.
Hi @GeoffNN ,
I am facing a similar issue, were you able to re-solve this?
Yep, I’m getting the same with resnet50 from torchvision
torch vesion: ‘1.8.1+cu102’
torchvision version: ‘0.9.1+cu102’
I was using the
wandb library with
wandb.watch(model). This was causing the problem, so I removed this part.
Thanks a lot @GeoffNN. I too was using wandb.watch and on removing it now it goes through. You saved my day!
Just curious, how did you think of the “watch” to be the reason?
wandb.watch was logging the gradients, so had to call
backward(), which seemed related to backpropagating twice in the graph.
Were you also using
Yes I was actually! Downgrading to torchvision 0.8.2 actually fixed the issue.
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/torch_stable.html
Think there is something in torchvision 0.9.0 that doesn’t play nice with wandb.watch? Just searched and found this issue on
wandb too incase you hadn’t seen it
I did try to downgrade but got CUDA Out of Memory error, so didn’t take that route. Somehow, didn’t come across this link though I was searching in all directions