Is it possible to give a lower learning rate on specific weight of specific layer?

autograd
1

Is it possible to give a lower learning rate on specific weight of specific layer?
e.g.)
if the weight index list = [1,2,3,4,5]
and give convolution layer 5 weight index list to 0.1 learning rate
and give other convolution layer 5 weight to 0.01

Is it possible?

2

I think you can do it by defining two optimizers and passing the desired params to each one with different learning rates:

params1 = ...
params1 = ...
opt1 = torch.optim.Adam(params1, lr1)
opt2 = torch.optim.Adam(params2, lr2)

or you can also do it like this, from pytorch tutorials here:

optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)

This means that model.base ’s parameters will use the default learning rate of 1e-2 , model.classifier ’s parameters will use a learning rate of 1e-3 , and a momentum of 0.9 will be used for all parameters.

1 Like
3

Thanks! I’ll give it a try :clap: