Optimizer doesn't work but manually updating weights does

1

I implemented a custom layer and I’m having trouble getting the optimizer to work. I create my optimizer

optimizer = optim.SGD(net.parameters(), lr=0.01)

and my training loop which consists of this

optimizer.zero_grad()  
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()

doesn’t update the weights. I print the weights after initializing the model, then train, and print the weights again and they are the same. I also printed the gradients during training and they are nonzero.

However when I run

learning_rate = 0.01
for f in net.parameters():
    f.data.sub_(f.grad.data * learning_rate)

it does update the weights. Does anyone know why this is happening?

1 Like
2

Hi,

How and when do you create the optimizer?
Do you change the weights after creating the optimizer by any chance?

1 Like
3

I just realized I wasn’t updating the optimizer with the new model in between runs. Thanks!

1 Like