# Custom Optimizer in PyTorch

**URL:** <https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397>\
**Category:** Uncategorized\
**Created:** [August 3, 2018, 5:07pm UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397 "2018-08-03T17:07:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![artix41](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/artix41/32/5967_2.png) [@artix41](https://discuss.pytorch.org/u/artix41)\
**Post date:** [August 3, 2018, 5:07pm UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/1 "2018-08-03T17:07:50Z")

</div>

For a project that I have started to build in PyTorch, I would need to implement my own descent algorithm (a custom optimizer different from RMSProp, Adam, etc.). In TensorFlow, it seems to be possible to do so ([https://towardsdatascience.com/custom-optimizer-in-tensorflow-d5b41f75644a](https://towardsdatascience.com/custom-optimizer-in-tensorflow-d5b41f75644a)) and I would like to know if it was also the case in PyTorch.

I have tried to do it by simply adding my descent vector to the leaf variable, but PyTorch didn’t agree: “a leaf Variable that requires grad has been used in an in-place operation.”. When I don’t do the operation in-place, the “new” variable loses its position of leaf, so it doesn’t work neither…

Is there an easy way to create such a custom optimizer in PyTorch?

Thanks in advance 🙂

---

<div class="post-metadata">

**Author:** ![Latope2-150](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ed8c4c/32.png) [@Latope2-150](https://discuss.pytorch.org/u/Latope2-150)\
**Post date:** [August 3, 2018, 6:28pm UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/2 "2018-08-03T18:28:06Z")

</div>

You may want to look at that post: [Regarding implementation of optimization algorithm](https://discuss.pytorch.org/t/regarding-implementation-of-optimization-algorithm/20920) 🙂

---

<div class="post-metadata">

**Author:** ![artix41](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/artix41/32/5967_2.png) [@artix41](https://discuss.pytorch.org/u/artix41)\
**Post date:** [August 3, 2018, 6:40pm UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/3 "2018-08-03T18:40:06Z")

</div>

Thanks for pointing it out, I will try to do what they suggest 🙂

---

<div class="post-metadata">

**Author:** ![tux](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tux/32/2364_2.png) [@tux](https://discuss.pytorch.org/u/tux)\
**Post date:** [August 4, 2018, 12:54am UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/4 "2018-08-04T00:54:32Z")

</div>

Hi Artix !

You can write your own update function for sure !  
To update your weights, you might use the optimiser library. But you  
can also do it yourself. For example, you can basically code the  
gradient descent, the SGD or Adam using the following code.

> net = NN()  
> learning\_rate = 0.01  
> for param in net.parameters():  
> weight\_update = smth\_with\_good\_dimensions  
> param.data.sub\_(weight\_update \* learning\_rate)

As you can see, you have access to your parameters in net.parameters(), so you can update them like you want.

If you want more specific examples, you can go here where I implemented both SVRG and SAGA : [GitHub - kilianFatras/variance\_reduced\_neural\_networks: Implementation of SVRG and SAGA optimization algorithms for deep learning topics.](https://github.com/kilianFatras/variance_reduced_neural_networks) (variance reduced algorithmes) ! If you have any further question, do not hesitate 🙂

---

<div class="post-metadata">

**Author:** ![artix41](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/artix41/32/5967_2.png) [@artix41](https://discuss.pytorch.org/u/artix41)\
**Post date:** [August 4, 2018, 9:38am UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/5 "2018-08-04T09:38:30Z")

</div>

Thank you so much, that’s exactly what I needed 😀 !

---

<div class="post-metadata">

**Author:** ![111150](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111150/32/15863_2.png) [@111150](https://discuss.pytorch.org/u/111150)\
**Post date:** [September 12, 2019, 7:58am UTC](https://discuss.pytorch.org/t/custom-optimizer-in-pytorch/22397/6 "2019-09-12T07:58:22Z")

</div>

Is these code is useful for image classification also?