What difference are there in using an optimizer ( torch.optim.ChosenOptimizer) to vary learning rates as opposed to an lr_scheduler (torch.optim.lr_scheduler.ChosenScheduler )?
What are some possible consequences of using both?
Hi,
You can choose any optimizer, for ex: SGD or ADAM and create your own learning rate schedules. Else if you want to use an off the shelf scheduler, you have some choices like ReduceLROnPlateau, ExponentialLR etc.
I think the later would be a better choice.
1 Like
Additionally to what @kira explained, note that the lr_scheduler just adjusts the learning rate. It does not update the weights of your model. You would still have to create and call an optimizer.
Usually the scheduler is also called once in every epoch, while the optimizer is called for every mini-batch.
2 Likes