# Weight decay in the optimizers is a bad idea (especially with BatchNorm) **URL:** \ **Category:** Uncategorized\ **Created:** [April 26, 2018, 12:37am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994 "2018-04-26T00:37:34Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Michael\_Oliver](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_oliver/32/94_2.png) [@Michael\_Oliver](https://discuss.pytorch.org/u/Michael_Oliver)\ **Post date:** [April 26, 2018, 12:37am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/1 "2018-04-26T00:37:34Z")
Correct me if I’m wrong, but there is no reason the beta and gamma parameters in BatchNorm should ever be subject to weight decay, ie L2 regularization, that pulls them toward 0. In fact it seems like a very bad idea to pull them toward 0. I know you can use Per-parameter options to get around the optimizers default behavior, but it seems like bad default behavior, especially with how commonly batchnorm is used. It seems better to only specify regularization where you want it rather than have the default apply it everywhere and thus even places it shouldn’t. I believe Pytorch could actually a cleaner regularization framework, perhaps similar to what we implemented in Keras. Thoughts? ---
**Author:** ![Chuong98](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chuong98/32/12271_2.png) [@Chuong98](https://discuss.pytorch.org/u/Chuong98)\ **Post date:** [May 29, 2019, 2:35am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/2 "2019-05-29T02:35:51Z")
It has been a year, but no feedback on it ? I absolutely agree with @Michael_Oliver . For example, in the official example training on ImageNet [https://github.com/pytorch/examples/tree/master/imagenet](https://github.com/pytorch/examples/tree/master/imagenet), it seems to me that Batch Norm Weight and Bias are added to Regularization Loss. I think the correct way to implement it shoulde be: ```auto optimizer = torch.optim.SGD( model.parameters(), args.lr, momentum=args.momentum) # ,weight_decay=args.weight_decay) #Remove weight decay in here cls_loss = criterion(output, target) reg_loss = 0 for name,param in model.named_parameters(): if 'bn' not in name: reg_loss += torch.norm(param) loss = cls_loss + args.weight_decay*reg_loss #Manual add weight decay ``` Please confirm or help with more elegant solutions. Thank you. ---
**Author:** ![rwightman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/32.png) [@rwightman](https://discuss.pytorch.org/u/rwightman)\ **Post date:** [May 29, 2019, 4:05pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/3 "2019-05-29T16:05:31Z")
I usually create a fn like add\_weight\_decay below. In current form it will add all batch norm parameters and bias layers to the no\_decay list. I use this instead of looking for ‘bn’ strings in the name because that isn’t always consistent from model to model. It’s usually suggested that bias params should also not be decayed, so this does the job for me. You can still use the name if it works for your case though. Separating it otherwise would be a pain. ```auto def add_weight_decay(model, weight_decay=1e-5, skip_list=()): decay = [] no_decay = [] for name, param in model.named_parameters(): if not param.requires_grad: continue if len(param.shape) == 1 or name in skip_list: no_decay.append(param) else: decay.append(param) return [ {'params': no_decay, 'weight_decay': 0.}, {'params': decay, 'weight_decay': weight_decay}] ``` When I create the optimizer, I put this block in front (usually all this is wrapped in a optim creation factory that also picks the optimizer to create from config or cmd args… ```auto weight_decay = args.weight_decay if weight_decay and filter_bias_and_bn: parameters = add_weight_decay(model, weight_decay) weight_decay = 0. else: parameters = model.parameters() if args.opt.lower() == 'sgd': optimizer = optim.SGD( parameters, lr=args.lr, momentum=args.momentum, weight_decay=weight_decay, nesterov=args.nesterov) ... ``` ---
**Author:** ![Chuong98](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chuong98/32/12271_2.png) [@Chuong98](https://discuss.pytorch.org/u/Chuong98)\ **Post date:** [May 30, 2019, 2:18am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/4 "2019-05-30T02:18:13Z")
> [@rwightman](#): > > (usually all this is wrapped in a optim creation factory that also picks the optimizer to create from config or cmd args… Nice. Thank you. But I am just curious why even the official training code still ignores this fundamental error. ---
**Author:** ![huangbiubiu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huangbiubiu/32/10560_2.png) [@huangbiubiu](https://discuss.pytorch.org/u/huangbiubiu)\ **Post date:** [September 5, 2019, 2:58am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/5 "2019-09-05T02:58:23Z")
Will the weight decay on bn layers affect the performance of the model? Why? ---
**Author:** ![Chuong98](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chuong98/32/12271_2.png) [@Chuong98](https://discuss.pytorch.org/u/Chuong98)\ **Post date:** [September 15, 2019, 7:20am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/6 "2019-09-15T07:20:36Z")
Well, Weight decay basically pulls the norm of paramters to 0. In Batch norm, e.g x\_hat = (x -beta)/gamma, you don’t want beta and gamma go to 0. Otherwise, BN is meaningless and erroneous. ---
**Author:** ![fgoldmann](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fgoldmann/32/20212_2.png) [@fgoldmann](https://discuss.pytorch.org/u/fgoldmann)\ **Post date:** [February 4, 2020, 10:35am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/7 "2020-02-04T10:35:56Z")
> [@Michael\_Oliver](#): > > Correct me if I’m wrong, but there is no reason the beta and gamma parameters in BatchNorm should ever be subject to weight decay, ie L2 regularization, that pulls them toward 0. In fact it seems like a very bad idea to pull them toward 0. This finding is also backed by this publication: > **[L2 Regularization versus Batch and Weight Normalization](https://arxiv.org/abs/1706.05350)** > > Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no... While I very much appreciate the code snippet provided by @rwightman , I too would like to see at least an option in PyTorch to exclude normalization layers from weight decay. ---
**Author:** ![Hzzone](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hzzone/32/11183_2.png) [@Hzzone](https://discuss.pytorch.org/u/Hzzone)\ **Post date:** [July 9, 2020, 6:58am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/8 "2020-07-09T06:58:03Z")
Please refer to the code of optimizer in PyTorch. In detail, after backward, the weight will be added to the grad of weight~(L2 weight decay). We could also directly use the above solution to avoid apply weight decay to bn. However, I have another more elegant method like function below: ```python def apply_weight_decay(*modules, weight_decay_factor=0., wo_bn=True): ''' https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/5 Apply weight decay to pytorch model without BN; In pytorch: if group['weight_decay'] != 0: grad = grad.add(p, alpha=group['weight_decay']) p is the param; :param modules: :param weight_decay_factor: :return: ''' for module in modules: for m in module.modules(): if hasattr(m, 'weight'): if wo_bn and isinstance(m, torch.nn.modules.batchnorm._BatchNorm): continue m.weight.grad += m.weight * weight_decay_factor ``` Please note that it should be applied after `loss.backward`. Besides, as far as I know, all bn layers in pytorch are inherited from `_BatchNorm` class. ---
**Author:** ![JIALI\_MA](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiali_ma/32/25079_2.png) [@JIALI\_MA](https://discuss.pytorch.org/u/JIALI_MA)\ **Post date:** [February 17, 2021, 4:15pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/9 "2021-02-17T16:15:19Z")
I think you may have mistaken beta and gamma with the running mean and running variance. The correct bn operation should be: x\_hat = [(x-running\_mean)/running\_var]\*gamma + beta So the beta and gamma are actually the bias and weight in bn layer. ---
**Author:** ![mMagmer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/32/43283_2.png) [@mMagmer](https://discuss.pytorch.org/u/mMagmer)\ **Post date:** [October 24, 2021, 7:43pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/10 "2021-10-24T19:43:07Z")
@rwightman, i’m no expert, but as @JIALI_MA suggested, the running averages of batch norm layer are buffers not parameters, hence they can’t be in `model.parameters()` ```auto import torch import torch.nn as nn # With Learnable Parameters m = nn.BatchNorm2d(5) ``` ```auto print(dict(m.named_parameters())) ``` output: ```auto {'weight': Parameter containing: tensor([1., 1., 1., 1., 1.], requires_grad=True), 'bias': Parameter containing: tensor([0., 0., 0., 0., 0.], requires_grad=True)} ``` ```auto print(dict(m.named_buffers())) ``` output: ```auto {'running_mean': tensor([0., 0., 0., 0., 0.]), 'running_var': tensor([1., 1., 1., 1., 1.]), 'num_batches_tracked': tensor(0)} ``` ---
**Author:** ![rwightman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/32.png) [@rwightman](https://discuss.pytorch.org/u/rwightman)\ **Post date:** [October 24, 2021, 8:51pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/11 "2021-10-24T20:51:10Z")
batch norm has learnable affine weight and bias parameters separate from the running mean and var ---
**Author:** ![bilzrd](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bilzrd/32/42518_2.png) [@bilzrd](https://discuss.pytorch.org/u/bilzrd)\ **Post date:** [July 26, 2022, 11:18pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/12 "2022-07-26T23:18:21Z")
The following paper[1] shows experimental results that “the effect of regularization was concentrated on the BN layer,” and we should be cautious about making the behavior such that the weight decay of the BN layer is off by default. > As evidence, we found that almost all of the regularization effect of weight decay was due to > applying it to layers with BN (for which weight decay is meaningless). It seems that the mechanism of weight decay is not fully understood even in the research field. At least until there is a clear empirical and theoretical basis, the above modification should be withheld. [1] [[1810.12281] Three Mechanisms of Weight Decay Regularization](https://arxiv.org/abs/1810.12281) ---
**Author:** ![turian](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/turian/32/30217_2.png) [@turian](https://discuss.pytorch.org/u/turian)\ **Post date:** [August 29, 2022, 2:35am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/13 "2022-08-29T02:35:56Z")
I would also be interested in a simple option for weight decay not to apply to batchnorm. fast.ai, citing Tencent, [reports that removing weight decay for batchnorm was helpful](https://www.fast.ai/2018/08/10/fastai-diu-imagenet/). ---
**Author:** ![addisonklinke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/addisonklinke/32/79137_2.png) [@addisonklinke](https://discuss.pytorch.org/u/addisonklinke)\ **Post date:** [November 3, 2022, 1:55pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/14 "2022-11-03T13:55:50Z")
> [@rwightman](#): > > I use this (i.e. `len(param.shape) == 1`) instead of looking for ‘bn’ strings in the name because [the name] isn’t always consistent from model to model The intended logic seems more readable to me when using `isinstance()`. Here’s an `add_weight_decay()` variant of @Hzzone’s `apply_weight_decay()` ```python def add_weight_decay( model, weight_decay=1e-5, skip_list=(nn.InstanceNorm, nn.BatchNorm)): """Using .modules() with an isinstance() check""" decay = [] no_decay = [] for module in model.modules(): params = [p for p in module.parameters() if p.requires_grad] if isinstance(module, skip_list): no_decay.extend(params) else: decay.extend(params) return [ {'params': no_decay, 'weight_decay': 0.}, {'params': decay, 'weight_decay': weight_decay}] ``` ---
**Author:** ![Equim](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/equim/32/55162_2.png) [@Equim](https://discuss.pytorch.org/u/Equim)\ **Post date:** [November 25, 2022, 9:43am UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/15 "2022-11-25T09:43:51Z")
However, I found another research stating that weight decay is good for BN in some network architectures. > Unlike weight decay on weights in e.g. convolutional layers, which typically directly precede nor- > malization layers, weight decay on γ and β can have a regularization effect so long as there is a > path in the network between the layer in question and the ultimate output of the network, as if such > paths do not pass through another normalization layer, then the weight decay is never “undone” by > normalization. > … > Regularization in the form of weight decay on the normalization parameters γ and > β can be applied to any normalization layer, but is only effective in architectures with particular > connectivity properties like ResNets and in tasks for which models are already overfitting. in [https://arxiv.org/pdf/1906.03548.pdf](https://arxiv.org/pdf/1906.03548.pdf) ---
**Author:** ![TriKri](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/9de0a6/32.png) [@TriKri](https://discuss.pytorch.org/u/TriKri)\ **Post date:** [January 12, 2023, 12:09pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/16 "2023-01-12T12:09:51Z")
What you implement here is not L2 regularization. For L2 regularization, you want to add the squared norm, nor the norm itself. (I have seen that sometimes you also divide the squared norm by 2) Also, are you sure that the test `'bn' not in name` is sufficient for determining whether a parameter belongs to a batch normalization layer? Otherwise, I think your approach looks good. ---
**Author:** ![TriKri](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/9de0a6/32.png) [@TriKri](https://discuss.pytorch.org/u/TriKri)\ **Post date:** [January 12, 2023, 1:14pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/17 "2023-01-12T13:14:34Z")
> in [https://arxiv.org/pdf/1906.03548.pdf](https://arxiv.org/pdf/1906.03548.pdf) “He et al. (2019) encourage disabling weight decay on γ and β, but ultimately find diminished performance by doing so.” Interesting. For ResNets, the only parameters that it makes sense to regularize using L2 regularization is essentially the γ parameters of the last batch normalization layer in each block. L2 regularizing any other parameter doesn’t have any effect (as long as you L2 regularize all of them in tandem)—except making the parameters smaller in magnitude and thus more sensitive to change, which actually may mess with the learning rate in a big way—see the paper [On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay](https://arxiv.org/abs/2106.15739). ---
**Author:** ![niceblue](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niceblue/32/57381_2.png) [@niceblue](https://discuss.pytorch.org/u/niceblue)\ **Post date:** [February 15, 2023, 10:09pm UTC](https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/18 "2023-02-15T22:09:28Z")
Absolutely : “why even the official training code still ignores this fundamental error” Essentially this means the (in a lot of cases, requiring any norm in particular) weight decay is completely broken “out of the box” in Pytorch. i.e. you cannot just set a weight decay parameter in the optimiser and expect it to work. I had a hard time even finding this forum when I hit this problem. I was very puzzled why AdamW did not work, and weight decay did not work. Only in a harder-to-find forum like this is it discussed at all. It would seem very straightforward to allow all norm layers and biases to be weight decay of zero in the optimiser code. It doesn’t even need to be the default setting (as this could break existing code). It can be a additional option as already mentioned here already (4 years ago …)