Correct way to use(order) scheduler in pytorch 1.6

2
  1. Yes, use the pattern recommended by the documentation.
  2. No.

The background is that people found out that it doing the LR step first gives unintuitive maths w.r.t. how/when the learning rate changes, so it is strictly recommended to do the LR step last. As always, advice on the internet may be outdated (seems to be the case here, you could help out with filing an issue or PR on their github) or even bad advice (probably not the case here, @ritchieng has been around for a long time and does know his stuff pretty well in my experience, but so I sometimes found stack overflow had really strange recommendations).

Regarding the warning: This might be some more general configuration about warnings on your system. If you believe it is because of PyTorch, you could double-check the versions (torch.__version__) and see in torch.optim to see which path it uses and see if lr_scheduler.py has the warning.

Best regards

Thomas

2 Likes