Torch.no_grad() makes any difference

Hello to everyone.

I am traying my model inference time using:

model.eval()
with torch.no_grad():
       y = model(x)

VS

model.train()
y = model(x)

These two operations take exactly the same time. Is it normal or am I making something wrong?

Thanks in advance.

Hi,

The eval mode will make a difference only if you use special Modules that behave differently in eval mode like dropout or batchnorm. But even in that case, the runtime might not change that much.
The no_grad mode disables the autograd so it will make a significant difference in memory usage but should not change much for runtime (as we work hard to make the autograd light so that it can run at every forward).

1 Like