# Different optimization process when put parameters in different param groups?

**URL:** <https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859>\
**Category:** Uncategorized\
**Created:** [June 24, 2019, 7:32pm UTC](https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859 "2019-06-24T19:32:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunshineatnoon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshineatnoon/32/13446_2.png) [@sunshineatnoon](https://discuss.pytorch.org/u/sunshineatnoon)\
**Post date:** [June 24, 2019, 7:32pm UTC](https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859/1 "2019-06-24T19:32:45Z")

</div>

Here is a minimal example to explain my question, suppose we have an optimizer:

```auto
optimizer = optim.Adam(model.parameters(), lr = args.lr)

```

and another one that put parameters in different parameter groups but using same learning rate:

```auto
optimizer = optim.Adam([{'params': model.aaa},
                        {'params': param for name, param in model.named_parameters() if 'aaaa' not in name}],
                        lr = args.lr)

```

In my experiments, these two optimizers give different results. Specifically, the `aaa` parameter in the first case gets a small gradient (doesn’t change much during training) while it gets larger gradients in the second case. But if I understand right, these two optimizers don’t have any difference except putting parameters into different param groups. So why are they behaving differently here?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [June 25, 2019, 5:24am UTC](https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859/2 "2019-06-25T05:24:22Z")

</div>

that sounds like a bug, but i cant fathom why.

Because the implementation of Adam is simple, and it just loops over param groups: [https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L60](https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L60)

---

<div class="post-metadata">

**Author:** ![sunshineatnoon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshineatnoon/32/13446_2.png) [@sunshineatnoon](https://discuss.pytorch.org/u/sunshineatnoon)\
**Post date:** [June 25, 2019, 4:17pm UTC](https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859/3 "2019-06-25T16:17:15Z")

</div>

Thanks for your reply. I found if I manually add all parameters instead of using `for name param in model.named_parameters()`, then it works fine.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 25, 2019, 4:26pm UTC](https://discuss.pytorch.org/t/different-optimization-process-when-put-parameters-in-different-param-groups/48859/4 "2019-06-25T16:26:28Z")

</div>

Could there be a typo `model.aaa` vs `'aaaa'`?
