About LR range Test y OneCycleScheduler

[Jona (Jonathan Muñoz)]

Hi everyone,

I read the paper about LR range test

1506.01186.pdf

And the one about OneCyclePolicy

1708.07120.pdf

I copied this implementation of the LR range test

How Do You Find A Good Learning Rate

I’ve also documented myself about de OneCyclePolicy

The 1cycle policy

https://towardsdatascience.com/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6

It seems to be something many people wonder:

Have I implemented implemenation of learning rate finder correctly?

How to use Pytorch OneCycleLR in a training loop (and optimizer/scheduler interactions)?

https://mc.ai/super-convergence-with-just-pytorch/

How does one use torch.optim.lr_scheduler.OneCycleLR()?

LR parameter in optimizer larger than max_lr in OneCycleLr

Hello, I made a little mistake in my code (I think at least …), where I put the following : # Optimizer wd = 0.01 optimizer = optim.Adam(model.parameters(), lr=10e-4, weight_decay=wd) # LR scheduler scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-4, steps_per_epoch=len(train_loader), epochs=epochs, pct_start=0.3) # only available on pytorch v1.3.0 Notice that in the optimizer I put a learning rate of 10e-4, whereas I put the max_lr on OneCycleLR to 1e-4. So in this situation w…

Suppose I have implemented correctly a LR Test

• How is the correct orchestation between the optimizer and the one cycle scheduler? Because, the optimizer has the param lr, and the scheduler max_lr?
• Can somebody provide me some example?, since I’ve been looking for it with no success

Thanks