How do I apply weight decay (L2) selectively?

My model is based on CNN and LSTM.
What I want to do is to apply L2 regularization to LSTM only.

However, As I know, in optim, it seems there no way to apply weight seperately.

Is there any way that I can try?? :slight_smile:

1 Like

Momentum and such are handled by the Optimizer itself, but as far as I know, weight decay, such as L1 and L2, can be implemented as a separate step, after the optimizer step?

So, seems like you could just grab the parameter Tensors/Variables for your LSTM, and subtract a fraction of the L2 norm from them?