Questions about updating weight

Hi, I have some question about backward() function.
As I understand, the backward() function only updates weight values that affect a lot to the loss.
In this case, many weight values that do not affect to the loss are only do minor update or skip.
I want to see the change in loss by updating all weight values, not just updating the weight values that have a large impact on the loss.
Please tell me how to update all weight values.

Thank you so much.

Gradient descent updates all of the weights. But some get more updated than others because they contributed most to the loss on the data given during that batch.

Various optimizers approach the problem of gradient descent updates differently, but all that Iā€™m aware of evaluate and update all of the weights during each backward pass.

Thanks to you, it is easy to understand for me.
Then I have another question. Can I adjust the degree of updates for weights?
I want to reduce the update degree of weights that have a large impact on loss, and increase the update degree of other weights.

You could just use clamp on the gradients. For example:

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(100, 128), nn.ReLU(), nn.Linear(128, 10))

data = torch.rand(128, 100)

labels = torch.rand(128, 10)

criterion = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

outputs = model(data)

loss = criterion(outputs, labels)
loss.backward()

# run this after loss.backward() and before optimizer.step()
for param in model.parameters():
    print(param.grad)
    param.grad = torch.clamp(param.grad, min = -0.001, max=0.001)
    print(param.grad)

optimizer.step()

Thank you so much!
It is very helpful to me.

1 Like