What's the difference between scheduler.step() and scheduler.step(epoch)?

1

I’m using 1.4.0. What’s the difference between scheduler.step() and scheduler.step(epoch)? What does
scheduler.step(epoch) do?

import torch

optimizer1 = torch.optim.SGD([torch.randn(1, requires_grad=True)], lr=1e-3)
exp_lr_scheduler1 = torch.optim.lr_scheduler.MultiStepLR(optimizer1,
                                             milestones=[5, 10], gamma=0.1)

optimizer2 = torch.optim.SGD([torch.randn(1, requires_grad=True)], lr=1e-3)
exp_lr_scheduler2 = torch.optim.lr_scheduler.MultiStepLR(optimizer2,
                                             milestones=[5, 10], gamma=0.1)

for epoch in range(1, 15):
    exp_lr_scheduler1.step()
    exp_lr_scheduler2.step(epoch)
    print('Epoch {}, lr1 {}, lr2 {}'.format(epoch,
        optimizer1.param_groups[0]['lr'],
        optimizer2.param_groups[0]['lr']))

The result is like follows:

Epoch 1, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 2, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 3, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 4, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 5, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 6, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 7, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 8, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 9, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 10, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 11, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 12, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 13, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 14, lr1 1e-05, lr2 1.0000000000000003e-05

They have different behavious when using MultiStepLR.

1 Like
2

Thanks, it worked for me, and I also tried it in torch 1.4.0. The results are just like your results.

But it’s worth noting that when this code was run in torch 1.2.0, the results were as follows

Epoch 1, lr1 0.001, lr2 0.001
Epoch 2, lr1 0.001, lr2 0.001
Epoch 3, lr1 0.001, lr2 0.001
Epoch 4, lr1 0.001, lr2 0.001
Epoch 5, lr1 0.0001, lr2 0.0001
Epoch 6, lr1 0.0001, lr2 0.0001
Epoch 7, lr1 0.0001, lr2 0.0001
Epoch 8, lr1 0.0001, lr2 0.0001
Epoch 9, lr1 0.0001, lr2 0.0001
Epoch 10, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 11, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 12, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 13, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 14, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05