PyTorch has new functionality
torch.inference_mode as of v1.9 which is “analogous to
torch.no_grad… Code run under this mode gets better performance by disabling view tracking and version counter bumps.”
If I am just evaluating my model at test time (i.e. not training), is there any situation where
torch.no_grad is preferable to
torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn’t surface as a runtime error then I assume it is indeed preferable to use
More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.
I asked a similar question elsewhere but it was the wrong forum.