Cyclic Learning rate - How to use

autograd
2

Your usage of the scheduler is generally fine and the warning is thrown, as we are patching the optimizer.step method in a similar way in apex as is done by the scheduler in this line of code.

To avoid this warning, initialize the scheduler after running amp.initialize(model, optimizer, opt_level).

Also, if you want, you could also add this check to avoid changing the learning rate, if the optimization step was skipped due to a gradient overflow:

    optimizer.step()
    if amp._amp_state.loss_scalers[0]._unskipped != 0: # assuming you are using a single optimizer
        scheduler.step()
1 Like