Dear experts,

Is it possible to use Adam for a set of parameters of the model and for an other set of parameters to use SGD for instance?

Thanks

Dear experts,

Is it possible to use Adam for a set of parameters of the model and for an other set of parameters to use SGD for instance?

Thanks

Yes, you just need to set several optimizers passing the corresponding parameters and everything else is the same.

Well, so imagine I define

optim1= SGD(set1 of model parameter)

optim2= Adam(set2 of model parameter)

what about the scheduler, can I define two of them like

sched1=StepLR(optim1)

sched2=ReduceLROnPlateau(optim2)

Thanks

Yes, each LR has a parameter called param_groups. Basically each group of parameters you pass to the each optimizer will be taken as a group. Schedulers work over these groups.