# Print current learning rate of the Adam Optimizer? **URL:** \ **Category:** Uncategorized\ **Created:** [March 20, 2018, 10:23am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204 "2018-03-20T10:23:11Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![flupadup](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e56c9b/32.png) [@flupadup](https://discuss.pytorch.org/u/flupadup)\ **Post date:** [March 20, 2018, 10:23am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/1 "2018-03-20T10:23:12Z")
At the beginning of a training session, the Adam Optimizer takes quiet some time, to find a good learning rate. I would like to accelerate my training by starting a training with the learning rate, Adam adapted to, within the last training session. Therefore, I would like to print out the current learning rate, Pytorchs Adam Optimizer adapts to, during a training session. thanks for your help 🙂 ---
**Author:** ![lelouedec](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lelouedec/32/3563_2.png) [@lelouedec](https://discuss.pytorch.org/u/lelouedec)\ **Post date:** [March 20, 2018, 10:25am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/2 "2018-03-20T10:25:02Z")
> for param\_group in optimizer.param\_groups: > print(param\_group[‘lr’]) should do the job ---
**Author:** ![jpeg729](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/32/3464_2.png) [@jpeg729](https://discuss.pytorch.org/u/jpeg729)\ **Post date:** [March 20, 2018, 10:51am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/3 "2018-03-20T10:51:51Z")
There is a paper that proposes tuning the learning rate using the gradient of the update rule with respect to the learning rate itself. Basically, it dynamically learns the learning rate during training. If you are interested, here is the paper [https://arxiv.org/abs/1703.04782](https://arxiv.org/abs/1703.04782) and my implementation [https://github.com/jpeg729/pytorch\_bits/blob/master/optim/adam\_hd.py#L67](https://github.com/jpeg729/pytorch_bits/blob/master/optim/adam_hd.py#L67) ---
**Author:** ![flupadup](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e56c9b/32.png) [@flupadup](https://discuss.pytorch.org/u/flupadup)\ **Post date:** [March 20, 2018, 12:47pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/4 "2018-03-20T12:47:21Z")
thanks for the quick response, but unfortunately, this command just prints out the initial learning rate and not the current adapted one. I would really surprise me, if it wasn’t possible, get the adapted learning rate somehow… ---
**Author:** ![flupadup](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e56c9b/32.png) [@flupadup](https://discuss.pytorch.org/u/flupadup)\ **Post date:** [March 20, 2018, 12:47pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/5 "2018-03-20T12:47:59Z")
thank you, I will look into it! ---
**Author:** ![lelouedec](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lelouedec/32/3563_2.png) [@lelouedec](https://discuss.pytorch.org/u/lelouedec)\ **Post date:** [March 20, 2018, 12:48pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/6 "2018-03-20T12:48:46Z")
this method print the currently used learning rate by the optimizer. ---
**Author:** ![flupadup](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e56c9b/32.png) [@flupadup](https://discuss.pytorch.org/u/flupadup)\ **Post date:** [March 20, 2018, 12:58pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/7 "2018-03-20T12:58:06Z")
at the moment I am just using two input images for my training. When I start a training session with the network, pretrained by me, the error increases by some magnitudes (from a few hundred to 10.000 up to 40.000) and commutes than back to the level, where it was at the end of the last session. Through all this the learning rate printed out on the console is always the same, initial one, what makes ne sense to me. I don’t know what else could be the reason for this big temporal fluctuation of the error. ---
**Author:** ![jpeg729](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/32/3464_2.png) [@jpeg729](https://discuss.pytorch.org/u/jpeg729)\ **Post date:** [March 20, 2018, 1:27pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/8 "2018-03-20T13:27:14Z")
Did you save the optimizer state with the model? Different optimizers tend to find different solutions so changing optimizers or resetting their state can perturbe training. That is why it can be important to not only save the model parameters but also the optimizer state. Of course, in this case, that might have nothing to do with the fluctuation of the error that you are seeing. ---
**Author:** ![Jonathan\_R\_Williford](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jonathan_r_williford/32/5606_2.png) [@Jonathan\_R\_Williford](https://discuss.pytorch.org/u/Jonathan_R_Williford)\ **Post date:** [November 8, 2018, 2:08am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/9 "2018-11-08T02:08:46Z")
Adam has a separate learning rate for each parameter. The `param_group['lr']` is a kind of base learning rate that does not change. There is no variable in the PyTorch Adam implementation that stores the dynamic learning rates. One could save the optimizer state, as mentioned here: > [@Saving and loading a model in Pytorch?](https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610): > > If I have a model class and a trainer class. I create an instance of the model and train it. model = mymodel() train = trainer.train(model...) How can I save the model to a file, after it has been trained and how can I then reload it and continue training? I searched for this but didn’t get an answer. The PyTorch implementation of Adam can be found here: [https://pytorch.org/docs/stable/\_modules/torch/optim/adam.html](https://pytorch.org/docs/stable/_modules/torch/optim/adam.html) The line `for p in group['params']:` iterates over all the parameters and calculates the learning rate for each parameter. I found this helpful: [http://ruder.io/optimizing-gradient-descent/index.html#adam](http://ruder.io/optimizing-gradient-descent/index.html#adam) ---
**Author:** ![henrique](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henrique/32/10098_2.png) [@henrique](https://discuss.pytorch.org/u/henrique)\ **Post date:** [April 27, 2019, 4:56pm UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/10 "2019-04-27T16:56:25Z")
> [@Jonathan\_R\_Williford](#): > > dynamic learning rates Sorry guys, but these comments seem very misleading! Optimizers have a fixed learning rate for all parameters. `param_group['lr']` would allow you to set a different LR for each layer of the network, but it’s generally not used very often, and most people have 1 single LR for the whole nn. What Adam does is to save a running average of the gradients for each parameter (not a LR!). The learning rate is still the same throughout the whole training, unless you use a lr\_scheduler like CosineAnnealingLR etc. ---
**Author:** ![Mithun\_Paul](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mithun_paul/32/13591_2.png) [@Mithun\_Paul](https://discuss.pytorch.org/u/Mithun_Paul)\ **Post date:** [September 20, 2019, 2:21am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/11 "2019-09-20T02:21:24Z")
also you might want to look into : ReduceLROnPlateau in [lrScheduler](https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html) ---
**Author:** ![Prasad\_Raghavendra](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prasad_raghavendra/32/24424_2.png) [@Prasad\_Raghavendra](https://discuss.pytorch.org/u/Prasad_Raghavendra)\ **Post date:** [March 22, 2025, 1:56am UTC](https://discuss.pytorch.org/t/print-current-learning-rate-of-the-adam-optimizer/15204/12 "2025-03-22T01:56:55Z")
Then, it is not the comments that are misleading, but, the library itself. It should have something like LR\_view or something like that (at the very least). I am doing some optimiser research and I am having a pretty difficult time trying to get some LR out.