Inplace addition for l2 regularization

1

Suppose I need to add a l2 regularization term to my loss, and its calculated like below

regularizer = torch.tensor(0.)
for name, param in model.named_parameters():
     if 'weight' in name:
         regularizer += torch.sum(torch.norm(param, dim=0))

I have two questions
one is that, for the initilization of ‘regularizer’, do I need to set the ‘requires_grad=True’

regularizer = torch.tensor(0., requires_grad=True)

Second one is that, is the inplace addition appropriate here? Do I need to modify it to

regularizer = regularizer  + torch.sum(torch.norm(param, dim=0))
2
  1. No, you don’t need to use requires_grad attribute and could even create the initial regularizer variable as a Python scalar: regularizer = 0.
  2. As long as Autograd doesn’t raise an error, the inplace operation should work fine. In case you are concerned about scripting the model (and loss calculation), you could check if removing the inplace ops would allow more operator fusions, but that might be an edge case.
1 Like