How to specify different learning rates for bias and weight?

vision
1

Hi, I am fairly new to pytorch and I am trying to implement a basic CNN. One of the things that I dont know how to do is set different learning rates for each layer and set independent learning rates for each parameter (weight, bias).

My attempt is:

optim.Adam([{'params': net.conv1.weight, 'lr': 1e-3}, {'params':net.conv1.bias, 'lr': 1e-4}])

Is this the right approach to setting independent learning rates for parameters?

1 Like
2

Yes. If you have multiple conv layers, you can also construct a list of weights, and a list of biases, and use one param group for each.

2 Likes