# Don't Regularize Me, Bro

**URL:** <https://discuss.pytorch.org/t/dont-regularize-me-bro/4946>\
**Category:** autograd\
**Created:** [July 16, 2017, 3:11pm UTC](https://discuss.pytorch.org/t/dont-regularize-me-bro/4946 "2017-07-16T15:11:16Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![uapatira](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/uapatira/32/670_2.png) [@uapatira](https://discuss.pytorch.org/u/uapatira)\
**Post date:** [July 16, 2017, 7:02pm UTC](https://discuss.pytorch.org/t/dont-regularize-me-bro/4946/4 "2017-07-16T19:02:54Z")

</div>

Thanks for the tips, guys. Your suggestions, and looking at the `.step()` method of [an optimizer](http://pytorch.org/docs/master/_modules/torch/optim/sgd.html#SGD.step), provide some key insight. Here’s a compact example for reference purposes, in case anyone else has this need:

```
model = nn.Sequential(
  nn.Conv2d(...),
  nn.PReLU(...),
  nn.Conv2d(...),
  nn.PReLU(...),
  nn.Conv2d(...),
)

params = [
  { 'params' : model[0].parameters() },
  { 'params' : model[2].parameters() },
  { 'params' : model[4].parameters() },

  { 'params' : model[1].parameters(), 'weight_decay':0 },
  { 'params' : model[3].parameters(), 'weight_decay':0 },
]

```

Then…

`opt = optim.SGD(params, lr=0.002, weight_decay=1e-5)`

Using two separate optimizers might be problematic due to this line: `p.data.add_(-group['lr'], d_p)`

---

_[View the full topic](https://discuss.pytorch.org/t/dont-regularize-me-bro/4946)._