Hello Torch Community,
I have a question about the usage of detach() during validation of the model. As we know, detach() creates a new tensor that shares storage with its original one, but does not requires grad. If the validation of the model is done inside with torch.no_grad() scope, do we need to use detach() in that case ?
Thanks in advance.
Inside the no_grad context manager, the autograd essentially “looks away” so no expensive gradient computation is performed and you should be fine using it mostly.
That said, there are certain cases in which tensors even when created within no_grad require gradients. (requires_grad isn’t False)
That said, you might want to dig into these cases more and choose whether to use detach or not. Otherwise, no_grad works for evaluating.
PS. Also don’t forget to call model.eval()
