As far as I can tell, there is no built-in class in torch.optim.lr_scheduler that can automatically do linear learning rate warm-up. I know this would be quite easy to implement, EG by subclassing torch.optim.lr_scheduler.LRScheduler. But considering linear learning rate warm up is used in some of the most all-time highly cited ML papers (EG “Attention is all you need”), and there are several existing forum posts + SE questions on this topic, it would make sense if this was included in torch.optim.lr_scheduler by default.