Only training multiple_choice_head of my output model while leaving the rest of the weights untouched

nlp
1

Hello,
If I set my optimizer and scheduler in the following way, will it prevent the weights for the model_gpt2.transformer.h.parameters() to not change during the training, while ensuring the weights from model_gpt2.multiple_choice_head.parameters() to change during the training? I don’t want the model_gpt2.transformer.h.parameters() to be altered during the training, I only want to adjust model_gpt2.multiple_choice_head.parameters() with my training loop:

optimizer_1 = torch.optim.SGD(model_gpt2.transformer.h.parameters(), lr=0.0)
optimizer_2 = torch.optim.Adam(model_gpt2.multiple_choice_head.parameters(), lr=0.00013)
scheduler_1 = torch.optim.StepLR(optimizer_1, step_size=1, gamma=1.5)
scheduler_2 = torch.optim.StepLR(optimizer_2, step_size=1, gamma=1.5)

Thank you,

2

Instead of modifying the optimizer like that, a better way of doing it would be:

for param in model_gpt2.transformer.h.parameters():
    param.requires_grad = False
for param in model_gpt2.multiple_choice_head.parameters():
    param.requires_grad = True
1 Like