Is there a Pytorch implementation of RAdam + LookAhead?

1

I was looking at the new paper by Geoffrey Hinton and was wondering if it’s been implemented. Look at the article:

Cross posted:

2

Here: https://github.com/mgrankin/over9000

3

RAdam+LA

from radam import RAdam
from optimizer import Lookahead
base_optim = RAdam(model.parameters(),lr = 0.001) 
optimizer =  Lookahead(base_optim, k=5, alpha=0.5)

RAdam | LAMB | Ralamb | Lookahead

2 Likes
4

The RAdam link seems broken (a 404 on github - maybe it’s not public?)

Anyway, this link works.