Pass all parameters to optimizer instead of only nonfrozen parameters

autograd
1

I’m curious what happens in this scenario. If we set requires_grad = False to some parameters, but accidentally pass these to the optimizer, are they skipped over or still optimized? As in the following:

model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, 2)

model_conv = model_conv.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)

Note that model_conv.parameters() should actually be model_conv.fc.parameters().

Are all of model_conv parameters optimized or only model_conv.fc?

2

Hi,

Assuming no gradients were computed for them before and their .grad field is None to begin with. Then the optimizer will just ignore them because they don’t have any gradient (as the backward won’t populate them).