Implementing new Optimization algorithms

I am looking forward to implement ‘Quasi Newton algorithm’ in pytorch.

Looking for suggestions of other optimization algorithms that you think are useful? I will go through and implement bunch of it.

Last year, quite a few people talked about KFAC, but it might not be too easy to fit into the pytorch optimizer framework.
One of the most prominent implementations seems to be this:

Best regards

Thomas

I think for Regression Problems the Levenberg Marquardt Algorithmus is quite usefull!