LARS LAMB optimizers

1

I searched inside https://pytorch.org/docs/master/optim.html but haven’t found any Layer-wise AdaptiveRate Scaling optimizer LARS implemented. Same for LAMB. Are there any plans soon? Should we add FR?

2

Hi,
LARC is not available directly in PyTorch but it is in Apex https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py

1 Like