Understanding step lr schedule

James_W · September 5, 2021, 9:20am

I am looking at some code from fb here:

facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_moco.py#L372

    
      
                  entries = [self.prefix + self.batch_fmtstr.format(batch)]
                  entries += [str(meter) for meter in self.meters]
                  print('\t'.join(entries))
          
          
    def _get_batch_fmtstr(self, num_batches):
                  num_digits = len(str(num_batches // 1))
                  fmt = '{:' + str(num_digits) + 'd}'
                  return '[' + fmt + '/' + fmt.format(num_batches) + ']'
          
          

          
def adjust_learning_rate(optimizer, epoch, args):
              """Decay the learning rate based on schedule"""
              lr = args.lr
              if args.cos:  # cosine lr schedule
                  lr *= 0.5 * (1. + math.cos(math.pi * epoch / args.epochs))
              else:  # stepwise lr schedule
                  for milestone in args.schedule:
                      lr *= 0.1 if epoch >= milestone else 1.
              for param_group in optimizer.param_groups:
                  param_group['lr'] = lr

which uses stepwise lr scheduler as follows (ignoring cos lr scheduler):

def adjust_learning_rate(optimizer, epoch, args):
    """Decay the learning rate based on schedule"""
    lr = args.lr
    for milestone in args.schedule:
        lr *= 0.1 if epoch >= milestone else 1.
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

and the corresponding scheduler is this:

parser.add_argument('--schedule', default=[120, 160], nargs='*', type=int,
                    help='learning rate schedule (when to drop lr by 10x)')

my understanding from the above is that at epochs 120 and 160, the learning rate should be dropped by 10x.

However, I think what these lines do:

    for milestone in args.schedule:
        lr *= 0.1 if epoch >= milestone else 1.

is to drop lr at 121,122,123… It kind of does not make sense to me or am I missing something here? I think it should be:

    for milestone in args.schedule:
        lr *= 0.1 if epoch == milestone else 1.

I wonder if I am missing something fundamental here?

James_W · September 5, 2021, 11:04am

apologies. I figured out why. I read the code wrong - it is using the lr from the args to slash the lr in epochs!