Recommended learning rate scheduler for Adam

1

Hi,
I’m trying to train an LSTM network, and using Adam as optimizer.
What is the recommended learning rate scheduler to use, that usually fits best to Adam?
I was under the impression that Adam controls the learning rate internally, but I see that if I manually reduce the learning rate when the validation loss reaches a plateau, I manage to further reduce the loss.

Thanks,
Guy

2

To my best of my knowledge, it depends on the model you are training.
Personally, I decay it by 0.1 if the validation loss rise.
BTW, it seems RMSprop is used more often in LSTM.

3

Hi @guyrose3, How are you reducing the learning rate while using Adam? I’m trying to do a similar process, but it seems that PyTorch’s implementation of Adam does not account for an external learningrate scheduler.

I’ve started a thread on Stack Exchange. See this for more info: