Regularization in Torch

Hi, I just transfer from Tensorflow to Pytorch. One quick question about the regularization loss in the Pytorch,

Does Pytorch has something similar to Tensorflow to calculate all regularization loss automatically?

Or we need to implement it by ourselves?

1 Like

if you simply want to use it in optimization, you can use keyword weight_decay of torch.optim.Optimizer.

Thanks, if I want to output the penalty, for example l2 loss. Do you think the following workable, as a simple implemetation?

def get_reg_loss(model):
     reg_loss = 0

     for param in model.parameters():
         reg_loss += param**2

    _lambda = 0.001
    reg_loss += _lambda * reg_loss
    return reg_loss

I think it’s ok, but notice that it will also penalt bias.
maybe it is better to use named_parameters, it’s added in 0.1.12

import torch
import torch.nn as nn
import torch.optim as optim
m = nn.Sequential(
      nn.Linear(10, 20),
      nn.Linear(20, 20),
weights, biases = [], []
for name, p in m.named_parameters():
   if 'bias' in name:
       biases += [p]
       weights += [p]

  {'params': weights},
  {'params': biases, weight_decay=0}
], lr=1e-2, momentum=0.9, weight_decay=1e-5)


Thank you very much. Does the pretrained model has this functionality?

	model = models.resnet18(pretrained=True)

	num_ftrs = model.fc.in_features
	model.fc = nn.Linear(num_ftrs, 2)
	model = model.cuda()

	weights, biases = [], []
	for name, p in model.named_parameters():
		if 'bias' in name:
			biases += [p]
			weights += [p]

This will gives me a bug:
AttributeError: 'ResNet' object has no attribute 'named_paramenters'

you have a typo