Hi there,
I was wondering if someone could shed some light on the following questions:
Why is ReduceLROnPlateau the only object without get_lr() method among all schedulers?
How to retrieve the learning rate in this case? Previously without scheduler I would do optimizer.param_groups[0]['lr'] but now after using the scheduler and printing optimizer.param_groups[0]['lr'] I see no change in the learning rate?
How to save the optimizer state if you’re using a scheduler. Previous examples I’ve seen show optimizer.state_dict() in order to save the optimizer state should we replace that with scheduler.state_dict()?
The scheduler might track multiple param_groups as described here.
It should still work as shown in this example:
model = nn.Linear(10, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
optimizer, patience=10, verbose=True)
for i in range(25):
print('Epoch ', i)
scheduler.step(1.)
print(optimizer.param_groups[0]['lr'])
No, you should still save the optimizer’s state_dict (and also call optimizer.step(), as the scheduler is not a replacement for the optimizer). Additionally, you could also store the scheduler’s state_dict.
By the way, is there anyway to obtain the learning rate from the ReduceLROnPlateau scheduler?
When I’m using other schedulers they all have get_last_lr() function; but since ReduceLROnPlateau is not inherited from _LRScheduler this function is not defined. Any equivalent way with get_last_lr() with this scheduler?
You could use the internal scheduler._last_lr attribute, the scheduler.state_dict() or alternatively you could check the learning rate in the optimizer via optimizer.param_groups[0]['lr'].
Note that the first two approaches would only work after the first scheduler.step() call.