Different model reset strategies

Hi,
I tried to find more information on

def weight_reset(m):
    if isinstance(m, conv1D):
        m.reset_parameters()
model.apply(weight_reset)

but I couldn’t find any documentation on that. What exactly is happening(will the model weights revert to definition weights?) and how is it different from weight init function on all layers?

def weight_init(m):
    if isinstance(m, conv1D)
        nn.init.xavier_uniform_(m.weight)
model.apply(weight_init)

The reset_parameter method is defined for each module (or its base class), as seen here for your conv layer.

The difference between both codes is, that the first one will use the default initialization which was specified for the particular layer, while the second approach will use your defined initialization (in your example xavier_uniform).

1 Like