Implementing new Optimization algorithms

I am looking forward to implement ‘Quasi Newton algorithm’ in pytorch.

Looking for suggestions of other optimization algorithms that you think are useful? I will go through and implement bunch of it.

Last year, quite a few people talked about KFAC, but it might not be too easy to fit into the pytorch optimizer framework.
One of the most prominent implementations seems to be this:

Best regards


I think for Regression Problems the Levenberg Marquardt Algorithmus is quite usefull!

HI, Filos
Do you know how to implement levenberg marquardt in pytorch?

Thanks !

You mean LM for general regression or as a substitute for SGD? Last one No, I’m sorry

Just LM for general regression. Do you have an example? Thanks!