I know that one can use something like torch.optim.lr_scheduler.LambdaLR to schedule lr.
Is there any way to schedule momentum hyperparameter, like in SGD for example?
With a bit of work, the pytorch schedulers can be extended to handle momentum as well. I’ve had a go implementing a version of LambdaLR that includes momentum, along with a few other scheduler types.
Is this the sort of thing you are looking for?
Alternatively, the pytorch/ignite project seems to cover this use case: https://github.com/pytorch/ignite
1 Like
A bit late to the post , but is there any advantage of using a momentum scheduler?
currently checking out Demon (paper: link) where the momentum starts at a very high value such as 0.9 and then reduces to a very low value towards the end.
Sometimes a combination of LR scheduler and momentum scheduler is useful I think , In other articles they suggest using low momentum for high learning rate and towards the end of the epoch increase the momentum and decrease the learning rate.
If there are advantages , Pytorch should include a feature for a momentum scheduler which we could add similar to lr_scheduler.step() we can add a momentum_scheduler.step(). A code example would be
lr_scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
momentum_scheduler = StepM(optimizer, step_size=30, gamma=0.1)
>>> for epoch in range(100):
>>> train(...)
>>> validate(...)
>>> lr_scheduler.step()
>>> momentum_scheduler.step()