How is optimizer connected to model

1

I am a little bit confused about the connection between the model and its optimizer. I am initializing the optimizer with the model’s parameters, e.g.:

optimizer = Adam(task_model.parameters(), lr=0.003)

When performing the optimizer step, how does the optimizer know which model to update?
I always was of the opinion that through initialization, the optimizer is somehow connected to the model. S.t. when performing an optimizer step, it will update the model’s parameter, meaning when checking the values of the model’s parameters (list(model.parameters())), the values should be different before and after performing the backward pass and optimizer step.
Is this correct? This also means that an optimizer is always tied to a specific model, correct?

2

Yes, your description is mostly correct.
You are explicitly passing the parameters via task_model.parameters() to the optimizer such that it would update these parameters using their .grad attribute (and potentially internal running stats depending on the optimizer) in its .step() call.

It’s not necessarily a model, as you can pass subsets of a model’s parameters or combinations of any parameter sets to the optimizer.

3

Thanks, now it’s clear to me! :slight_smile: