RuntimeError: Module backward hook for grad_input is called before the grad_output one. This happens because the gradient in your nn.M odule flows to the Module's input without passing through the Module's output

vision
1

I’m using a resnet18 from torchvision. When running the forward pass, I get the following warning:

UserWarning: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is deprecated and will be forbidden starting version 1.6. You can remove this warning by cloning the output of the custom Function.

And when I run the backward pass, I get

Module backward hook for grad_input is called before the grad_output one. This happens because the gradient in your nn.Module flows to the Module's input without passing through the Module's output. Make sure that the output depends on the input and that the loss is computed based on the output.

The error seems to stem from result = torch.relu_(input)
This didn’t happen previously.

2 Likes
2

Hi @GeoffNN ,

I am facing a similar issue, were you able to re-solve this?

1 Like
3

Yep, I’m getting the same with resnet50 from torchvision

torch vesion: ‘1.8.1+cu102’
torchvision version: ‘0.9.1+cu102’

4

I was using the wandb library with wandb.watch(model). This was causing the problem, so I removed this part.

1 Like
5

Thanks a lot @GeoffNN. I too was using wandb.watch and on removing it now it goes through. You saved my day!

Just curious, how did you think of the “watch” to be the reason?

6

Welcome! :slight_smile:

wandb.watch was logging the gradients, so had to call backward(), which seemed related to backpropagating twice in the graph.

7

Were you also using wandb?

8

Thanks @GeoffNN :+1:

9

Yes I was actually! Downgrading to torchvision 0.8.2 actually fixed the issue.

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/torch_stable.html

Think there is something in torchvision 0.9.0 that doesn’t play nice with wandb.watch? Just searched and found this issue on wandb too incase you hadn’t seen it

10

I did try to downgrade but got CUDA Out of Memory error, so didn’t take that route. Somehow, didn’t come across this link though I was searching in all directions :thinking: