Mixte optimizer

1

Dear experts,

Is it possible to use Adam for a set of parameters of the model and for an other set of parameters to use SGD for instance?

Thanks

2

Yes, you just need to set several optimizers passing the corresponding parameters and everything else is the same.

3

Well, so imagine I define
optim1= SGD(set1 of model parameter)
optim2= Adam(set2 of model parameter)
what about the scheduler, can I define two of them like
sched1=StepLR(optim1)
sched2=ReduceLROnPlateau(optim2)
Thanks

4

Yes, each LR has a parameter called param_groups. Basically each group of parameters you pass to the each optimizer will be taken as a group. Schedulers work over these groups.