Does a loss divided by n equivalent with learning rate / n?

autograd
1

Hello all, if my code is

n=10
learning_rate=1e-2
criterion= nn.BCEWithLogitsLoss()
optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-6)
loss= criterion(outputs, targets)
loss=loss / n
loss.backward()

Does the loss equivalent with the below by using learning rate / n?

n=10
learning_rate=1e-3  #1e-2 /n
criterion= nn.BCEWithLogitsLoss()
optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-6)
loss= criterion(outputs, targets)
loss.backward()

I am using SGD. IF not, what should I change in the first code without using loss=loss/n? Thanks

2

IF you’re using sgd, yes.

1 Like
3

Thanks for your reply. I have updated more detail in the question. Actually, I used SGD+momentum. Let me know if you want to change your answer with the updated question

4

If you’re using momentum and/or weight decay (or another optimizer) it won’t work.
For example SGD+weight decay (with learning rate lr and weight_decay wd will do the following update: w = w - lr * (dL/dw + wd*w). You can see that scaling your loss: dL/dw -> 1/n*dL/dw will not have the same effect as changing the learning rate lr -> 1/n*lr: the weight decay term will not be scaled the same way.

For other optimizer, you will need to check the formula of the update and see if scaling of the gradient and of the learning rate have the same effect on the update or not (it is really unlikely).

2 Likes
5

@albanD Please correct me if I am wrong. it seems like this is because of the way PyTorch’s SGD is different from other frameworks (e.g. caffe)

I am porting a network from caffe and am trying to understand why after/if I increase the lr (after certain epoch), network always becomes unstable (inf weights and nan loss).
~It seems like PyTorch’s SGD is more sensitive to lr changes because it is applied to velocity instead of gradients. Is there any particular reason for this choice?~
EDIT: I added a SGD that is more like other frameworks and still if I increase lr, the network becomes unstable. Decreasing lr is always fine.