Layer not present in model.parameters()

projects
1

I am trying to freeze all layers except the lm_head of the t5-base model:

model = AutoModelForSeq2SeqLM.from_pretrained('t5-base').to('cuda')
model.lm_head

Linear(in_features=768, out_features=32128, bias=False)

But when running the standard freezing loop, I don’t see lm_head in model parameters:

for param_name, param in model.named_parameters():
        if startswith_list(param_name, layers_keep_training):
            param.requires_grad = True
            params_to_train.append(param_name)
        else:
            param.requires_grad = False

Is it named differently, or nested? How would I freeze all layers except that one? If I freeze all named_parameters, might I miss anything like I’m missing lm_head here?

2

Indeed the lm_head attribute is missing so it seems the parameter might be initialized as another attribute.
Checking for the id shows this is the case:

w_id = id(model.lm_head.weight)
for name, param in model.named_parameters():
    if w_id == id(param):
        print(name)
# shared.weight

print((model.lm_head.weight == model.shared.weight).all())
# tensor(True)
1 Like