# With Adam optimizer, is it necessary to use a learning scheduler?

**URL:** <https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477>\
**Category:** Uncategorized\
**Created:** [January 13, 2020, 5:58am UTC](https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477 "2020-01-13T05:58:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hong\_Cheng](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hong_cheng/32/10639_2.png) [@Hong\_Cheng](https://discuss.pytorch.org/u/Hong_Cheng)\
**Post date:** [January 13, 2020, 5:58am UTC](https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477/1 "2020-01-13T05:58:07Z")

</div>

Adam can adapt its learning rate by the gradient updating. I think we may not need the learning rate scheduler.  
However, I worry that if with that kind of learning rate scheduler in Adam can jump out of the local minimal or get away from the local minimal

In [transfer\_learning\_tutorial](https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html), it use momentum SGD with a learning scheduler.

---

<div class="post-metadata">

**Author:** ![Eta\_C](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/32/17667_2.png) [@Eta\_C](https://discuss.pytorch.org/u/Eta_C)\
**Post date:** [January 13, 2020, 9:13am UTC](https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477/2 "2020-01-13T09:13:29Z")

</div>

Yes I have had such experience.  
Now in my project, I split `num_epochs` into three parts.

- `num_epochs_1` warm up.
- `num_epochs_2` Adam for speeding up covergence.
- `num_epochs_3` momentum SGD+CosScheduler for training.

My friend used Adam without learning rate scheduler in his project, and he found that the loss started to rise after some epochs.

You can find some discuss [here](https://stackoverflow.com/questions/39517431/should-we-do-learning-rate-decay-for-adam-optimizer). Although Adam can adaptively adjust the learning rate, but such ability is limited.

At least, for me, I think momentum SGD is the most stable optimizer and Adam/AdamW is a good tick to speed up covergence.

All these are my personal experiences. Is it necessary to use a learning scheduler? Maybe as the answer in the link says,

> It depends.

---

<div class="post-metadata">

**Author:** ![Aldebaran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aldebaran/32/23963_2.png) [@Aldebaran](https://discuss.pytorch.org/u/Aldebaran)\
**Post date:** [December 2, 2020, 4:19pm UTC](https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477/3 "2020-12-02T16:19:45Z")

</div>

Pytorch Adam algorithm implementation follows changes proposed in [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101) which states:

> Adam can substantially benefit from a scheduled learning rate multiplier. The fact that Adam  
> is an adaptive gradient algorithm and as such adapts the learning rate for each parameter  
> does not rule out the possibility to substantially improve its performance by using a global  
> learning rate multiplier, scheduled, e.g., by cosine annealing.

---

<div class="post-metadata">

**Author:** ![TMK04](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tmk04/32/62753_2.png) [@TMK04](https://discuss.pytorch.org/u/TMK04)\
**Post date:** [July 30, 2023, 7:14am UTC](https://discuss.pytorch.org/t/with-adam-optimizer-is-it-necessary-to-use-a-learning-scheduler/66477/4 "2023-07-30T07:14:46Z")

</div>

[Pytorch AdamW\* algorithm](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html)