Hello,
I use Adam Optimizer for training my network but when I print learning rate I realized that learning rate is constant and doesn’t change. Adam optimizer is adaptive learning rate algorithm but I don’t see such behavior in results.
if optimizer_name == 'SGD':
# optimizer
optimizer = torch.optim.SGD(
params=params,
lr=optimizer_arguments['lr'],
momentum=optimizer_arguments['momentum'],
weight_decay=optimizer_arguments['weight_decay'])
elif optimizer_name == 'Adam':
# Adam optimizer
optimizer = torch.optim.Adam(
params=params,
lr=optimizer_arguments['lr'],
weight_decay=optimizer_arguments['weight_decay'])
else:
raise ValueError('Optimizer name is incorrect!')