Does gradients change in Model.eval()?

autograd
1

I am confused between with torch.no_grad() and model.eval(). Does gradients change in model.eval() mode, or anyway to check that?

2

The gradient calculation is independent from the training mode in the model, which is changed via model.train() and model.eval().
To disable the gradient calculation, set the .requires_grad attribute of all parameters so False or wrap the forward pass into with torch.no_grad().
Calling model.eval() will change the behavior of some layers, such as nn.Dropout, which will be disabled, and nn.BatchNormXd, which will use the running stats during evaluation.

3 Likes
3

I have found this note useful in understanding the different ways one can disable gradients:

https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation

1 Like
4

Do you mean that in model.eval(), the parameters of the BN layer will continue to be updated, just using different statistics?

5

No, after calling model.eval() the running stats will be used to normalize the input activations and these running stats will not be updated in batchnorm layers using the default setup.