Is there a way to write my custom optimizer from scratch ?
The code for SGD is very readable given that you know how it works: https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html
You might ignore in-place operations they tend to use, and also ignore the “param_groups” if you don’t care.