Weight Decay Parameter

[julioeu99 (JCLopes)]

Hi, can someone explain me in newbie words (i´m new at deep learning word), what does the parameter weight decay on torch adam? And whats the impact if i change it from 1e-2 to 0.
Thank you.

[SANTOSH_S (SANTOSH)]

Hi
@julioeu99 weight decay in simple terms just reduces weights calculated with a constant(here 1e-2). This ensures that one does not have large weight values which sometimes leads to early overfilling.
Weight decay sometimes makes the model to converge slower.

By default pytorch has weight_decay=0

Some useful discussions on the same:

How does SGD weight_decay work?

The part that I circled doesn’t seem right to me: [39a9f417cb836820174d71e9a9318d5ccc6191f3] In L2 regularization, you modify the cost as follows [text_Regularized] The weight update should be then [text_updatedWeig] The way PyTorch applied the weight decay seems correct to me (you can drop the factor 2)

How pytorch implement weight_decay?

That line means, in other notation: d_p = d_p + weight_decay * p.data. Here’s a good article about why the L2 penalty is implemented by adding weight_decay * weight_i to the gradient: https://stats.stackexchange.com/questions/29130/difference-between-neural-net-weight-decay-and-learning-rate