Mixte optimizer

Dear experts,

Is it possible to use Adam for a set of parameters of the model and for an other set of parameters to use SGD for instance?


Yes, you just need to set several optimizers passing the corresponding parameters and everything else is the same.

Well, so imagine I define
optim1= SGD(set1 of model parameter)
optim2= Adam(set2 of model parameter)
what about the scheduler, can I define two of them like

Yes, each LR has a parameter called param_groups. Basically each group of parameters you pass to the each optimizer will be taken as a group. Schedulers work over these groups.