I am trying to freeze all layers except the lm_head of the t5-base model:
model = AutoModelForSeq2SeqLM.from_pretrained('t5-base').to('cuda')
model.lm_head
Linear(in_features=768, out_features=32128, bias=False)
But when running the standard freezing loop, I don’t see lm_head in model parameters:
for param_name, param in model.named_parameters():
if startswith_list(param_name, layers_keep_training):
param.requires_grad = True
params_to_train.append(param_name)
else:
param.requires_grad = False
Is it named differently, or nested? How would I freeze all layers except that one? If I freeze all named_parameters, might I miss anything like I’m missing lm_head here?