Hi, I just transfer from Tensorflow to Pytorch. One quick question about the regularization loss in the Pytorch,
Does Pytorch has something similar to Tensorflow to calculate all regularization loss automatically?
tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
Or we need to implement it by ourselves?
1 Like
if you simply want to use it in optimization, you can use keyword weight_decay
of torch.optim.Optimizer.
Thanks, if I want to output the penalty, for example l2 loss. Do you think the following workable, as a simple implemetation?
def get_reg_loss(model):
reg_loss = 0
for param in model.parameters():
reg_loss += param**2
_lambda = 0.001
reg_loss += _lambda * reg_loss
return reg_loss
I think it’s ok, but notice that it will also penalt bias.
maybe it is better to use named_parameters
, it’s added in 0.1.12
import torch
import torch.nn as nn
import torch.optim as optim
m = nn.Sequential(
nn.Linear(10, 20),
nn.ReLU(),
nn.Linear(20, 20),
nn.ReLU(),
)
weights, biases = [], []
for name, p in m.named_parameters():
if 'bias' in name:
biases += [p]
else:
weights += [p]
optim.SGD([
{'params': weights},
{'params': biases, weight_decay=0}
], lr=1e-2, momentum=0.9, weight_decay=1e-5)
6 Likes
Hi,
Thank you very much. Does the pretrained model has this functionality?
model = models.resnet18(pretrained=True)
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, 2)
model = model.cuda()
weights, biases = [], []
for name, p in model.named_parameters():
if 'bias' in name:
biases += [p]
else:
weights += [p]
This will gives me a bug:
AttributeError: 'ResNet' object has no attribute 'named_paramenters'
you have a typo
named_paramenters
->named_parameters