# Lower limit in learning rate in PyTorch torch.optim

I was going through the following information on reducing learning rates in PyTorch to really low value like 1e-9.

And I am amazed why doing ` loss = loss/100` is equivalent to reducing learning rate by 100? The full snippet is below.

``````outputs = model(batch)
loss = criterion(outputs, targets)

# Equivalent to lowering the learning rate by a factor of 100
loss = loss / 100

loss.backward()
self.optimizer.step()
``````

Thanks a lot

If you scale the loss, you’ll also scale the gradients.
In a simple use case, this can be used instead of changing the learning rate, as seen here:

``````# Setup
torch.manual_seed(2809)

lin = nn.Linear(2, 2, bias=False)
x = torch.randn(1, 2)

# Standard approach
out = lin(x)
loss = out.sum()
print(loss)
loss.backward()
tensor([[-1.1281,  0.8386],
[-1.1281,  0.8386]])

# loss scaling by x10
out = lin(x)
loss = out.sum() * 10
print(loss)
loss.backward()
tensor([[-11.2812,   8.3855],
[-11.2812,   8.3855]])

# loss scaling by x0.1