Oh. I was using Adam! Thanks for clarifying.
A not so thorough reading of, and lack of resolution in, old learning rate updation queries like this might be behind such confusion.
Maybe the optimizers where the internal state matters should be tagged at the torch optim page? Also, possibly at saving and loading models for inference. Thanks again for the prompt response.