Implementing new Optimization algorithms

bhushans23 · March 12, 2018, 8:50pm

I am looking forward to implement ‘Quasi Newton algorithm’ in pytorch.

Looking for suggestions of other optimization algorithms that you think are useful? I will go through and implement bunch of it.

tom · March 12, 2018, 9:13pm

Last year, quite a few people talked about KFAC, but it might not be too easy to fit into the pytorch optimizer framework.
One of the most prominent implementations seems to be this:

github.com

yaroslavvb/kfac_pytorch/blob/master/kfac_pytorch.py

import util as u
u.check_mkl()

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import numpy as np
import scipy
import sys

from torch.autograd.function import Function

def regularized_inverse(mat, lambda_=3e-3, inverse_method='numpy',
                        use_cuda=True):
  assert mat.shape[0] == mat.shape[1]
  ii = torch.eye(mat.shape[0])
  if use_cuda:
    ii = ii.cuda()

This file has been truncated. show original

Best regards

Thomas

Filos92 · July 8, 2019, 11:14am

I think for Regression Problems the Levenberg Marquardt Algorithmus is quite usefull!

wqsgsutd · March 6, 2020, 6:07am

HI, Filos
Do you know how to implement levenberg marquardt in pytorch?

Thanks !

Filos92 · March 6, 2020, 7:59am

You mean LM for general regression or as a substitute for SGD? Last one No, I’m sorry

wqsgsutd · March 13, 2020, 7:50am

Just LM for general regression. Do you have an example? Thanks!

fabiodimarco · December 2, 2024, 3:35am

For anyone still looking for a Levenberg-Marquardt implementation in PyTorch, I’ve developed one:

Hope it helps.