[Solved] Learning Rate Decay

vision
1

Hello I have seen some forum about Learning decay in pytorch for example in here . They said that we can adaptivelly change our learning rate in pytorch by using this code.

def adjust_learning_rate(optimizer, epoch):
        """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
         lr = args.lr * (0.1 ** (epoch // 30))
         for param_group in optimizer.param_groups:
             param_group['lr'] = lr

*) My question is has it been implemented in pytorch version 0.1 or 0.2 as default feature?, Or we must manually defined like the code? .
*) If we must manually defined like that function may I know your experiences the best epoch for dropping learning rate?, for example in that code is every 30 epoch.
-Thank you-

2 Likes
2

torch.optim.lr_scheduler module provides many different learning rate adjustment.
i think it would be the best practice way.

3 Likes
3

Have a look at http://pytorch.org/docs/master/optim.html#how-to-adjust-learning-rate
It is available starting from 0.2.0

You can still use the code you mentioned to adjust the learning rate as you want.

4 Likes
4

@Jing @trypag Thanks guys!, is there any best setting in parameter of epoch?, for example in the script it always using 30.

5

Poly rate scheduler is quite used at that time.

def poly_lr_scheduler(optimizer, init_lr, iter, lr_decay_iter=1,
                      max_iter=100, power=0.9):
    """Polynomial decay of learning rate
        :param init_lr is base learning rate
        :param iter is a current iteration
        :param lr_decay_iter how frequently decay occurs, default is 1
        :param max_iter is number of maximum iterations
        :param power is a polymomial power

    """
    if iter % lr_decay_iter or iter > max_iter:
        return optimizer

    lr = init_lr*(1 - iter/max_iter)**power
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

    return lr
8 Likes
6 
def adjust_lr(optimizer, epoch):
    lr = init_lr * (0.1 ** (epoch // 20))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

Remember invoke adjust_lr function at the beginning of each epoch.

3 Likes