Load optimizer for partial parameters

Suppose I have a model M with some parameters P. I train this using Adam and save the state_dict of the model and the optimizer. I now add few more parameters in the model to make it Mn and the parameters are Pn.

To load the variables from the partial model that is saved I do the following
state = Mn.state_dict() lstate = torch.load(model_path) state.update(lstate) Mn.load_state_dict(state)

This ensures that the old variables are loaded from the saved model and the new ones are in their initialized state.

If I try to do the same with the optimizer, it complains that the number of parameters in the new optimizer’s state_dict() is more than the loaded state dict.

What is the recommended method to restore the partial optimizer variables?

3 Likes

we haven’t really thought this through for the optimizer.
Best to either look into the optimizer internals (they’re not that complicated and they’re in python), or just construct a new optimizer.

I have been looking into the internals but its kind of tricky as the optimizer uses 'id’s, which change with each run, instead of parameters names as in the model state dict.
I guess for now the best/easiest solution is to make a new optimizer.

I have couple of follow-up questions about the optimizer I will ask in a separate post as they are more generic.