In this page, torch.nn.RMSNorm.
There’s a formula
It seems that this root sign should be removed? Because there’s already a root in RMS function.
It seems that this root sign should be removed? Because there’s already a root in RMS function.
Thanks for pointing this out. The issue is already fixed via PR #136727 and the current master docs also show the updated formula as seen here.