Kernel and Bias Regularization in nn.Linear()

1

Hi, Can anyone please guide how can we add the regularizer in the nn.Linear module same like tensorflow. In Tensorflow we add regularization in the Dense layer as:

tf.layers.dense(inputs=pool2_flat, units=2048, name='dense1', kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001), bias_regularizer=tf.contrib.layers.l2_regularizer(0.001))

Any thought how can add in nn.Linear?

Thank you.

2

Hi Mustansar!

Use the weight_decay constructor argument when you instantiate
your optimizer. For SGD (without momentum), it is the same as
L2 regularization. For Adam, it is not identical to L2 regularization,
but some people argue that Adam’s weight_decay works better
than L2 regularization. (Conceptually, they both play the same role.)

I don’t know the details of what tensorflow does, so I can’t say how
closely this might mimic the tensorflow implementation.

If you want to use weight_decay / L2 regularization with just some
layers, you can pass in parameter groups when you instantiate
your optimizer. Each group can contain different layers and have
different values for weight_decay, including 0.0 (which is its default
value).

Best.

K. Frank

1 Like