Does gradients change in Model.eval()?

I am confused between with torch.no_grad() and model.eval(). Does gradients change in model.eval() mode, or anyway to check that?

The gradient calculation is independent from the training mode in the model, which is changed via model.train() and model.eval().
To disable the gradient calculation, set the .requires_grad attribute of all parameters so False or wrap the forward pass into with torch.no_grad().
Calling model.eval() will change the behavior of some layers, such as nn.Dropout, which will be disabled, and nn.BatchNormXd, which will use the running stats during evaluation.

3 Likes

I have found this note useful in understanding the different ways one can disable gradients:

https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation

1 Like

Do you mean that in model.eval(), the parameters of the BN layer will continue to be updated, just using different statistics?

No, after calling model.eval() the running stats will be used to normalize the input activations and these running stats will not be updated in batchnorm layers using the default setup.