Simple L2 regularization?

19

Sorry for question here.
It is said that when regularization L2, it should only for weight parameters, but not bias parameters.(Is it right?:flushed:)
But the L2 regularization included in most optimizers in PyTorch, is for all of the parameters in the model (weight and bias).
least_squares_l2
I mean the parameters in the red box should be weight parameters only. (If what I heard of is right.)
How can I deal with it?

weight_p, bias_p = [],[]
for name, p in model.named_parameters():
  if 'bias' in name:
    bias_p += [p]
  else:
    weight_p += [p]

optim.SGD(
  [
    {'params': weight_p, 'weight_decay':1e -5},
    {'params': bias_p, 'weight_decay':0}
  ],
  lr=1e-2, momentum=0.9
)

Code here can deal with the problem above, is it right?:flushed::flushed:

1 Like