How to use torch.optim.lr_scheduler.ExponentialLR?

I am trying to train a LSTM model in a NLP problem.

I want to use learning rate decay with the torch.optim.lr_scheduler.ExponentialLR class, yet I seem to fail to use it correctly.
My code:

optimizer = torch.optim.Adam(dual_encoder.parameters(), lr = 0.001)

scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma = 0.95)

for epoch in range(200):

context_id_list, response_id_list, label_array = load_ids_and_labels(dataframe, word_to_id)

loss_acc = 0

for i in range(len(label_array)):

    context = autograd.Variable(torch.LongTensor(context_id_list[i]).view(len(context_id_list[i]),1), requires_grad = False)
    response = autograd.Variable(torch.LongTensor(response_id_list[i]).view(len(response_id_list[i]), 1), requires_grad = False)
    label = autograd.Variable(torch.FloatTensor(torch.from_numpy(np.array(label_array[i]).reshape(1,1))), requires_grad = False)
    score = dual_encoder(context, response)

    loss = torch.nn.functional.binary_cross_entropy_with_logits(score, label) 
    loss_acc +=[0]

    scheduler.step() #?
print("Epoch: ", epoch, ", Loss: ", (loss_acc/len(label_array)))

If I do it like this, according to one PyTorch example, the parameters do not get updated. If I use optimizer.step() instead, the scheduler will not even be applied, as far as I understand.

I would be happy about some code example!


You should add optimizer.step() into your training loop and move scheduler.step() into the epoch loop.


Seems to work, thanks!

What if I want to have the learning schedule defined on number of iterations instead of number of epochs? Can I still put by schedule.step() inside the inner loop?

Sure, you could guard it with a condition.
E.g. if you would like to call it every 100 iterations:

for epoch in range(epochs):
    for batch_idx, (data, target) in enumerate(data_loader):
        # Your training routine
        data = ...

        if (batch_idx+1) % 100 == 0:

Thanks! Here’s some more code…

        lstm = DSARNN(input_dims, sequence_length, cell_size)
        criterion = torch.nn.MSELoss()
        #optimiser = torch.optim.Adagrad(lstm.parameters(), lr=0.01)
        optimiser = torch.optim.SGD(lstm.parameters(), lr=0.1)
        scheduler = torch.optim.lr_scheduler.StepLR(optimiser, step_size=3, gamma=0.1)

        # register hooks


        # init Tensorboard
        tensorboard_step = 0
        writer = SummaryWriter(comment="LSTM Cell + input attention, entropy " + str(bc.entropy()).replace('.', '_'))

        for epoch in range(40):


            # train
            for minibatch, target in tqdm(Batches(train, target_input), total=len(train)):

                    output = lstm(minibatch)
                    loss = criterion(output, target)
                    tensorboard_step += 1
                    writer.add_scalar('training loss', loss, tensorboard_step)
                    writer.add_scalar('learning rate', get_learning_rate(optimiser), tensorboard_step)
                    loss = optimiser.step()
1 Like

scheduler.step() should be called after optimiser.step()

1 Like

thank you sooo much! :slight_smile:

1 Like

Welcome. Best of luck with your project.

1 Like

Use scheduler.step() instead of scheduler.step(epoch). It has strange behavious when using MultiStepLR. Though it works fine for StepLR in your example.


1 Like