I tried to use both adujusting lr methods, wanted to increase lr using lambdaLR from min_lr to max_lr(warm up), and then apply CosineAnnealingLR, but the learning rate seem not what I expected. it shows kind of ‘plateau’. Please kindly see below:
I attach my code below:
n_epochs = 10
# n_iter = len(train_loader)
n_iter = 28
max_lr = 1e-3; min_lr = 1e-5;
base_lr = max_lr / (n_epochs*n_iter)
bias = min_lr * (1/base_lr)
lr_ls = []
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
lambda1 = lambda x: bias+x
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1)
j = 0
for epoch in range(n_epochs):
for _ in range(n_iter):
j+=1
lr_ls.append(scheduler.get_last_lr()[0])
optimizer.step()
scheduler.step(j)
n_epochs = 50
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=200, eta_min=max_lr, last_epoch=j)
for epoch in range(n_epochs):
for _ in range(n_iter):
j+=1
lr_ls.append(scheduler.get_last_lr()[0])
optimizer.step()
scheduler.step()
plt.plot(lr_ls)
Actually, I guess I don’t totally understand CosineAnnealingLR param, last_epoch maybe.
Would you help me out?
I solved this problem by myself.
TL;DR: need to declare optimizer again with the last lr of previous optimizer
base_lr = optimizer.param_groups[0]['lr']
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
Please refer to the correction:
n_epochs = 10
# n_iter = len(train_loader)
n_iter = 28
max_lr = 1e-3; min_lr = 1e-5;
base_lr = max_lr / (n_epochs*n_iter)
bias = min_lr * (1/base_lr)
lr_ls = []
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
lambda1 = lambda x: bias+x
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1)
j = 0
for epoch in range(n_epochs):
for _ in range(n_iter):
j+=1
lr_ls.append(scheduler.get_last_lr()[0])
optimizer.step()
scheduler.step(j)
##################################### this part
base_lr = optimizer.param_groups[0]['lr']
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
#####################################
n_epochs = 50
#scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=200, eta_min=0, last_epoch=-1)
scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=200, T_mult=2, eta_min=min_lr)
for epoch in range(n_epochs):
for _ in range(n_iter):
j+=1
lr_ls.append(scheduler.get_last_lr()[0])
optimizer.step()
scheduler.step()
plt.plot(lr_ls, alpha=0.5)
plt.scatter(range(len(lr_ls)), lr_ls, s=0.3, c='r')