When i test my model, do I have to use model.eval() even though I am using 'with torch.no_grad() ?
These two have different goals:
model.eval()will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval model instead of training mode.
torch.no_grad()impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).
Derivative of two networks
Out of memory error during evaluation but training works fine!
Thank you very much for your quick and clear explanation.
Hey, this implies I should definitely do
"model.eval" while validating.
And, if memory and speed are not constraints;
"torch.no_grad()" can be ignored. Right?
Ahh with torch.no_grad() you’ll have much higher speeds and can use larger validation batch sizes so it’s useful if not recommended
Why do something that takes 5x more memory (the 5 here is for the example, not actual number in practice) and is slow, if you can just add one extra line to avoid it?
torch.no_grad() is actually the recommended way to perform validation !
yeah alright I meant to write “if not compulsory”
Different gradients under the same condition when generating adversarial examples
Thank you for the explanation!
torch.no_grad() also disable dropout layers?
So why is torch.no_grad() is not enabled by default inside model.eval() function? Is there a situation where we want to compute some gradients when in evaluation mode? Even if that’s the case, it seems like no_grad() method should be made an optional argument to eval(), and set to True by default.