# How to add a L2 regularization term in my loss function **URL:** \ **Category:** autograd\ **Created:** [May 3, 2018, 3:42am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411 "2018-05-03T03:42:04Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![JinChengWu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/32/4587_2.png) [@JinChengWu](https://discuss.pytorch.org/u/JinChengWu)\ **Post date:** [May 3, 2018, 3:42am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/1 "2018-05-03T03:42:04Z")
## Hi, I’m a newcomer. I learned Pytorch for a short time and I like it so much. ![%E5%9C%96%E7%89%87](https://discuss.pytorch.org/uploads/default/original/2X/9/90af8b995cfab96c5a4415217bc28ead3d08af31.png) I’m going to compare the difference between with and without regularization, thus I want to custom two loss functions. ```auto ###OPTIMIZER criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr = LR, momentum = MOMENTUM) ``` ## Can someone give me a further example? Thanks a lot! BTW, I know that the latest version of TensorFlow can support dynamic graph. But what is the difference of the dynamic graph between these two frameworks? ---
**Author:** ![Separius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/32/4165_2.png) [@Separius](https://discuss.pytorch.org/u/Separius)\ **Post date:** [May 3, 2018, 7:46am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/2 "2018-05-03T07:46:30Z")
set “weight\_decay” parameter to a non zero value in your optimizer(sgd, adam, …)(it’s the alpha in your equation) edit: I think it’s alpha times two actually ---
**Author:** ![JinChengWu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/32/4587_2.png) [@JinChengWu](https://discuss.pytorch.org/u/JinChengWu)\ **Post date:** [May 3, 2018, 7:51am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/3 "2018-05-03T07:51:13Z")
> [@JinChengWu](#): > > def backward I think I miss one row: def backward Cuz ‘w’ is the weight. It is updated continuously by steps. I just wonder if I need to do the grad-decent by myself? ---
**Author:** ![Separius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/32/4165_2.png) [@Separius](https://discuss.pytorch.org/u/Separius)\ **Post date:** [May 3, 2018, 7:53am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/4 "2018-05-03T07:53:18Z")
oh, in that case, iterate over your parameters (for p in self.parameters()) and add (p\*\*2).sum() to your loss ---
**Author:** ![JinChengWu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/32/4587_2.png) [@JinChengWu](https://discuss.pytorch.org/u/JinChengWu)\ **Post date:** [May 3, 2018, 8:01am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/5 "2018-05-03T08:01:23Z")
Can you give me a simple example, ex: MSE I’d like to know the mechanism of this custom class detaily.😱 ---
**Author:** ![Separius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/32/4165_2.png) [@Separius](https://discuss.pytorch.org/u/Separius)\ **Post date:** [May 3, 2018, 8:15am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/6 "2018-05-03T08:15:49Z")
have you seen [this](https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py)? there are two ways to handle backprop, doing it by hand or using the autograd package (and also a third way which is using both of them, by defining backward) if you are using the autograd, and your modules are composed of standard operations, you can simply define your loss without the L2 regularizer and in the optimizer define the regularizer ```auto class custom(nn.Module): def __init__ (self): super(). __init__ () def forward(self, x): return x net = custom() optimizer = optim.SGD(net.parameters(), lr=0.01, weight_decay=0.01) criterion = nn.MSELoss() for batch in batches: optimizer.zero_grad() y = net(batch['x']) loss = criterion(y, batch['y']) loss.backward() optimizer.step() ``` ---
**Author:** ![JinChengWu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/32/4587_2.png) [@JinChengWu](https://discuss.pytorch.org/u/JinChengWu)\ **Post date:** [May 3, 2018, 8:34am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/7 "2018-05-03T08:34:23Z")
Sure, I know this is custom neural network. I use the 3 layers CNN net defined by myself with the nn.MSELoss() before. It’s autograd. But now I want to compare the results if loss function with or without L2 regularization term. If I use autograd nn.MSELoss(), I can not make sure if there is a regular term included or not. p.s.:I checked that parameter ‘weight\_decay’ in optim means “add a L2 regular term” to loss function.👍 Furthermore, if I want to add a “L1” norm term in my loss function, I CANNOT USE THE autograd ? ---
**Author:** ![Separius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/32/4165_2.png) [@Separius](https://discuss.pytorch.org/u/Separius)\ **Post date:** [May 3, 2018, 9:02am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/8 "2018-05-03T09:02:02Z")
no, you can always use autograd (even if your function does not have a derivative, you can use something else as derivative and go backward from there), what i meant was that when you have simple functions, there is no need to write backward() yourself adding L1 loss is simple: ```auto loss = mse(pred, target) l1 = 0 for p in net.parameters(): l1 = l1 + p.abs().sum() loss = loss + lambda_l1 * l1 loss.backward() optimizer.step() ``` ---
**Author:** ![Separius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/32/4165_2.png) [@Separius](https://discuss.pytorch.org/u/Separius)\ **Post date:** [May 3, 2018, 9:06am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/9 "2018-05-03T09:06:27Z")
> [@JinChengWu](#): > > If I use autograd nn.MSELoss(), I can not make sure if there is a regular term included or not. > > p.s.:I checked that parameter ‘weight\_decay’ in optim means “add a L2 regular term” to loss function.👍 in general loss of a network has some terms, adding L2 term via optimizer class is really easy and there is no need to explicitly add this term (optimizer does it), so if you want to compare networks, you can simply tune weight\_decay ---
**Author:** ![derEitel](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/0ea827/32.png) [@derEitel](https://discuss.pytorch.org/u/derEitel)\ **Post date:** [October 24, 2018, 4:09pm UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/10 "2018-10-24T16:09:17Z")
I want to follow an implementation of a Keras model in which only on some conv layers an l2 `kernel_regularizer` has been used. Now I have followed your implementation but am wondering if it suffices to filter for the names of the layers that I want to include my regularization on or not. I.e. along the lines of: ```auto reg_lambda=0.01 l2_reg = 0 if isinstance(layer_names, list): for W in self.model.named_parameters(): if "weight" in W[0]: layer_name = W[0].replace(".weight", "") if layer_name in layer_names: l2_reg = l2_reg + W[1].norm(2) loss = loss + l2_reg * reg_lambda loss.backward() ``` ---
**Author:** ![fs\_z](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e9bcb4/32.png) [@fs\_z](https://discuss.pytorch.org/u/fs_z)\ **Post date:** [December 29, 2018, 8:33am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/11 "2018-12-29T08:33:36Z")
I have the same problem that use pytorch to achieve keras’s kernel\_regularizer. can you share this part code that have worked .THX ---
**Author:** ![mmdbrdrn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmdbrdrn/32/9966_2.png) [@mmdbrdrn](https://discuss.pytorch.org/u/mmdbrdrn)\ **Post date:** [June 10, 2019, 3:48pm UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/12 "2019-06-10T15:48:02Z")
weight decay doesn’t show good results with adam optimizer ---
**Author:** ![muammar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muammar/32/76367_2.png) [@muammar](https://discuss.pytorch.org/u/muammar)\ **Post date:** [June 14, 2019, 5:02pm UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/13 "2019-06-14T17:02:29Z")
> [@mmdbrdrn](#): > > weight decay doesn’t show good results with adam optimizer Why? Could you elaborate more on your reply? ---
**Author:** ![hemanth346](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hemanth346/32/20830_2.png) [@hemanth346](https://discuss.pytorch.org/u/hemanth346)\ **Post date:** [February 22, 2020, 1:45am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/14 "2020-02-22T01:45:16Z")
If someone’s landed up here for this, you can check out below answer on stackoverflow > ---
**Author:** ![Nasim\_Shafiee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nasim_shafiee/32/21085_2.png) [@Nasim\_Shafiee](https://discuss.pytorch.org/u/Nasim_Shafiee)\ **Post date:** [February 28, 2020, 4:24am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/15 "2020-02-28T04:24:29Z")
optimizer = optim.Adam(model.parameters(), lr=learning\_rate, weight\_decay=0.01) ---
**Author:** ![avh77](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a6a055/32.png) [@avh77](https://discuss.pytorch.org/u/avh77)\ **Post date:** [March 24, 2020, 12:03pm UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/16 "2020-03-24T12:03:38Z")
L2 regularizer and weight\_decay in Adam are a bit different thinks, more on that is available here: [https://openreview.net/pdf?id=rk6qdGgCZ](https://openreview.net/pdf?id=rk6qdGgCZ) ---
**Author:** ![Prajwal\_Sridhar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prajwal_sridhar/32/48040_2.png) [@Prajwal\_Sridhar](https://discuss.pytorch.org/u/Prajwal_Sridhar)\ **Post date:** [March 28, 2022, 9:04pm UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/17 "2022-03-28T21:04:51Z")
How do you experiment with different values for weight\_decay? So that you could show the amounts of regularization on the x axis and validation set performance on the y axis ---
**Author:** ![JINGHAN\_YANG](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinghan_yang/32/46872_2.png) [@JINGHAN\_YANG](https://discuss.pytorch.org/u/JINGHAN_YANG)\ **Post date:** [August 21, 2022, 3:55am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/18 "2022-08-21T03:55:40Z")
May I ask are you sure it is 2_alpha instead of 1/2_alpha? Where I can find this? ---
**Author:** ![Siladittya\_Manna](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siladittya_manna/32/26984_2.png) [@Siladittya\_Manna](https://discuss.pytorch.org/u/Siladittya_Manna)\ **Post date:** [November 1, 2023, 10:00am UTC](https://discuss.pytorch.org/t/how-to-add-a-l2-regularization-term-in-my-loss-function/17411/19 "2023-11-01T10:00:18Z")
I recently used this to add L2 regularization with LBFGS optimizer for a linear classification model ``` reg_loss = 0 for mod in self.model.modules(): if isinstance(mod, _BatchNorm): if self.decay_bn: for name, param in mod.named_parameters(recurse=False): reg_loss = reg_loss + param.norm(2) else: for name, param in mod.named_parameters(recurse=False): if not name.endswith("bias"): reg_loss = reg_loss + param.norm(2) else: if self.decay_bias: reg_loss = reg_loss + param.norm(2) ```