Parameter: weight decay- optimizer ADAM

[Mike2004 (Mike Long)]

Hello, can someone explain me better, what the weight decay parameter in optimizer ADAM, does?
Thank you.

[jmandivarapu1 (Jaya Krishna Mandivarapu)]

@Mike2004

someone explain me better, what the weight decay parameter in optimizer ADAM, does?
Thank you.

The weight_decay parameter adds a L2 penalty to the cost which can effectively lead to to smaller model weights.

How does SGD weight_decay work?

Hello, i write a toy code to check SGD weight_decay. but it seems to have no effect to the gradient update. am i misunderstand the meaning of weight_decay? thank you very much. PyTorch 1.0 import torch import numpy as np np.random.seed(123) np.set_printoptions(8, suppress=True) x_numpy = np.random.random((3, 4)).astype(np.double) w_numpy = np.random.random((4, 5)).astype(np.double) x_torch = torch.tensor(x_numpy, requires_grad=True) w_torch = torch.tensor(w_numpy, requires_grad=True) l…

https://dejanbatanjac.github.io/2019/07/02/Impact-of-WD.html

[Mike2004 (Mike Long)]

Thank you for you time, now i understand!