Can I use CosineAnnealingLR and ReduceLROnPlateau at the same time?

1

I want to implement a solution that slowly reduces LR when it is not in the plateau period, and quickly reduces LR when it is in the plateau period.

I firstly tried to use ChainedScheduler, but I got this error:

ValueError: ChainedScheduler does not support `ReduceLROnPlateau` scheduler as it requires additional kwargs to be specified when calling `step`, but got one at index 1 in the given schedulers sequence.

Then, I found these words in the documents of PyTorch 2.8.0:

Most learning rate schedulers can be called back-to-back (also referred to as chaining schedulers). The result is that each scheduler is applied one after the other on the learning rate obtained by the one preceding it.

So, I tried code like these:

import torch
from torch.optim.lr_scheduler import CosineAnnealingLR, ReduceLROnPlateau

num_epochs = 200  # to mock train 200 epochs 
iters = 100  # to mock there are 100 batches in each epoch

model = torch.nn.Sequential(torch.nn.Linear(16, 64), torch.nn.Linear(64, 1))
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)
cosine_scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs, eta_min=1e-5)
plateau_scheduler = ReduceLROnPlateau(optimizer, factor=0.5, patience=5, cooldown=2, min_lr=1e-5)

for epoch in range(num_epochs):
    for batch in range(iters):
        optimizer.step()

    if (50 <= epoch < 59) or (90 <= epoch < 102):
        validation_loss = 0.5  # to mock plateau period
    else:
        validation_loss = 1.0 / (epoch + 1)  # to mock loss decrease normally

    for group in optimizer.param_groups:
        lr = group["lr"]
        break
    print("{:03}\t{}".format(epoch, lr))

    cosine_scheduler.step()
    plateau_scheduler.step(validation_loss)

I asked some LLM models, and they didn’t recommend this approach, but they couldn’t give me a convincing reason.

However, the code output above seems to work fine.

But I haven’t seen any discussion of this. Can anyone tell me if this is the right approach?