`requires_grad` or `no_grad` in prediction phase

Based on the official tutorial, during prediction (after training and evaluation phase), we are supposed to do something like

model.eval()
with torch.no_grad():
    # run prediction

But let’s assume that we want to use state_dict extracted from trained model.
In this case, we usually do

model.load_state_dict(torch.load('path/to/state_dict'))
model.to(device)
model.eval()
with torch.no_grad():
  # run prediction

My question is, if there is no need for further training, is it possible to set requires_grad=False to all layers in model and run prediction without torch.no_grad(), or will there be any difference?

model.load_state_dict(torch.load('path/to/state_dict'))
for param in mode.parameters():
    param.requires_grad = False
model.to(device)
model.eval()
# run prediction
1 Like

Hi,

There will be a small difference in most cases I think.
If you set every requires_grad to False. After each op, it will check whether the output requires gradients.
If you set the no_grad mode, then it won’t even try.

1 Like

Hi, albanD!
Thanks for the reply.
I will use no_grad mode. :slight_smile:

If you have a prediction function as-

def predict(features):
    with torch.no_grad():
        return model(features)

Then it will always call this context-manager method, which in turn might affect inference performance.

So after training your model, I think it makes more sense to use “torch.set_grad_enabled(False)”, function since it disables any gradient computation globally, so then you can simple create your prediction function as-

def predict(features):
    return model(features)

Am I right @albanD?

1 Like

If you only ever want to do inference you can yes.
The context manager is nice because it restores the previous state and so is better when you use it to test in your training loop for example.