Scheduler for momentum?

I know that one can use something like torch.optim.lr_scheduler.LambdaLR to schedule lr.
Is there any way to schedule momentum hyperparameter, like in SGD for example?

With a bit of work, the pytorch schedulers can be extended to handle momentum as well. I’ve had a go implementing a version of LambdaLR that includes momentum, along with a few other scheduler types.

Is this the sort of thing you are looking for?

Alternatively, the pytorch/ignite project seems to cover this use case:

1 Like

A bit late to the post , but is there any advantage of using a momentum scheduler?

currently checking out Demon (paper: link) where the momentum starts at a very high value such as 0.9 and then reduces to a very low value towards the end.

Sometimes a combination of LR scheduler and momentum scheduler is useful I think , In other articles they suggest using low momentum for high learning rate and towards the end of the epoch increase the momentum and decrease the learning rate.

If there are advantages , Pytorch should include a feature for a momentum scheduler which we could add similar to lr_scheduler.step() we can add a momentum_scheduler.step(). A code example would be

lr_scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
momentum_scheduler = StepM(optimizer, step_size=30, gamma=0.1)
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     lr_scheduler.step()
>>>     momentum_scheduler.step()