Should the requires_grad=False of model weights when using torch.inference_mode

autograd
1

Hi,

As far as I understanding, when using the torch.inference_mode() or torch.no_grad(),
the model will not caculating the gradients and the requires_grad will be False.
I write a simple code to reproduce my question

import torch
import torch.nn as nn

class Network(nn.Module):
    def __init__(
        self,
    ):
        super().__init__()
        self.conv = nn.Conv2d(2, 32, (3, 3))

    def forward(self, x):

        with torch.inference_mode():
            self.eval()
            x = self.conv(x)
            print(x.requires_grad)
            print(self.conv.weight.requires_grad)

        return x

x = torch.randn(4, 2, 257, 513)
backbone = Network()
backbone(x)

The corresponding outputs:

False
True

The output requires_grad of x is False, but the requires_grad of self.conv.weight is True.
Is the behavior of the latter correct or there might be something that I haven’t noticed?
Thank you for your reply in advance.

2

This is expected since the inference_mode context won’t change the .requires_grad attributes of parameters or inputs. The output won’t be attached to a computation graph and no intermediate tensors will be stored, as expected.

1 Like
3

Ok, I got it. Thanks for your reply!

4

Quick clarifying question from a newbie. Does it mean that inference_mode would ignore all .requires_grad attributes on parameters or inputs? Would setting .requires_grad of parameters and inputs to False improve performance within inference_mode?

5

No, it won’t improve the performance as the gradient calculation is already disabled in inference_mode.

1 Like