How to Implement Custom Regularization Losses on the Weights?

[Eta_C]

Assume we want to customize L1 Loss Regularization.

An example for one parameter.

L1-regularization for a single matrix

You can add the L1 regularization to your loss and call backward on the sum of both. Here is a small example using the weight matrix of the first linear layer to apply the L1 reg: model = nn.Sequential( nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2) ) x = torch.randn(10, 10) target = torch.empty(10, dtype=torch.long).random_(2) criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(model.parameters(), lr=1e-3) for epoch in range(1000): optimizer.zero_grad() output = mode…

An example for each layer.

Simple L2 regularization?

Hi, The L2 regularization on the parameters of the model is already included in most optimizers, including optim.SGD and can be controlled with the weight_decay parameter as can be seen in the SGD documentation. L1 regularization is not included by default in the optimizers, but could be added by including an extra loss nn.L1Loss in the weights of the model. l1_crit = nn.L1Loss(size_average=False) reg_loss = 0 for param in model.parameters(): reg_loss += l1_crit(param) factor = 0.0005 lo…