test = torch.autograd.Variable(torch.randn([5,5]), requires_grad=True)
optimizer = torch.optim.Adam([test], lr = 0.0001)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)
lr_scheduler.get_lr()
This code snippet gives me a learning rate of 0.001
I want to start the optimizer with LR = 1e-4, and schedule a decay of 0.1 every 100 epochs.
It seems like the LR Scheduler starts me off at 1e-3 instead.
Can someone clarify?
This happens on Torch 0.3.1 and 0.4.0.
It’s a bit strange to see the initial learning rate seems to be too high, but it’s consistent with the example in the docs:
scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100):
scheduler.step()
train(...)
validate(...)
As you call scheduler first, the learning rate will be set to 1e-4 for the next step_size epochs.
test = torch.autograd.Variable(torch.randn([5,5]), requires_grad=True)
optimizer = torch.optim.Adam([test], lr = 0.0001)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)
lr_scheduler.step()
lr_scheduler.get_lr()
Yeah. The behaviour is weird. If I call step() before get_lr(), the output looks correct.
I am also finding the same issue while using lr scheduler. I want to start it from 1e-05 but it starts from 1e-03.
Another thing I found out that, if I increase the step size, then it will starts from 1e-04
Any one else got this issue. How they Solved it ?
Could you post a minimal code snippet to reproduce your issue?
base_lr = 0.000001 # Hyper parameters
max_lr = 1 # Hyper parameters
def main():
print(‘…Main Function starts…’)
optimizer = optim.SGD(model.parameters(), lr=base_lr, momentum=momentum)
scheduler = optim.lr_scheduler.CyclicLR(optimizer=optimizer,base_lr=base_lr,max_lr=max_lr,step_size_up=100,step_size_down=100)
def train(model, optimizer, scheduler,plot_loss,plot_lr, training_dataloader, batch_size, grids, grids_values, bounding_boxes, classes, lambda_coord, lambda_noobject):
model.train()
batch_loss = 0
for batch_index, batched_sample in enumerate(training_dataloader):
print(‘Current LR : {}’.format(scheduler.get_lr()))
batched_image = torch.tensor(batched_sample[‘image’], requires_grad=True, dtype=torch.float)
batched_label = torch.tensor(batched_sample[‘label’], requires_grad=True, dtype=torch.float)
batched_output = model(batched_image)
batched_output = batched_output.view(batch_size, grids, grids_values) # Convert the output size into [N X GRIDS(225 X 225) X (5 * B + C)]
loss = yolo_loss(batched_output, batched_label, grids, bounding_boxes, classes, lambda_coord, lambda_noobject)
batch_loss = batch_loss + loss.item()
optimizer.zero_grad()
loss.backward()
optimizer.step()
scheduler.step()
For this settings, the output is :
Changing the setting to step_size_up = 500 and nothing else, output is :
I might be misunderstanding the output, but the learning rate is starting from the base_lr=1e-6 in both screenshots.
Correct me, If I am wrong. It is showing at the first print base_lr = 1e-06, when the lr scheduler is not called. After calling lr scheduler, it jumps to 0.01 and scaling with (+0.01). Rather it should starts from 1e-06 after calling lr scheduler and should scale accordingly.
