Adam Optimizer and learning rate

Hello,

I use Adam Optimizer for training my network but when I print learning rate I realized that learning rate is constant and doesn’t change. Adam optimizer is adaptive learning rate algorithm but I don’t see such behavior in results.

    if optimizer_name == 'SGD':
        # optimizer
        optimizer = torch.optim.SGD(
            params=params,
            lr=optimizer_arguments['lr'],
            momentum=optimizer_arguments['momentum'],
            weight_decay=optimizer_arguments['weight_decay'])
    elif optimizer_name == 'Adam':
        # Adam optimizer
        optimizer = torch.optim.Adam(
            params=params,
            lr=optimizer_arguments['lr'],
            weight_decay=optimizer_arguments['weight_decay'])
    else:
        raise ValueError('Optimizer name is incorrect!')

Hi,

It is adaptive for each weight. The global lr does not change but the statistics for each weight does. You might want to double check the paper.