Why dropout changes weights?

1

I am trying to use nn.Dropout to drop some of the weights in trained network.
I expected this to result in zero-ing out some of the entries, but I also observe changes in numerical values:

params = [p for n, p in model.named_parameters() if 'features' not in n and len(p.shape) > 1]
for par in params:
    par = par.detach()
    par = torch.nn.Dropout2d(p=0.3)(par)

Before dropout:

tensor([[ 0.0022, 0.0041, 0.0004, …, -0.0026, -0.0028, -0.0021],
[ 0.0007, 0.0015, 0.0021, …, 0.0040, 0.0016, 0.0015],
[ 0.0007, 0.0005, 0.0021, …, -0.0049, -0.0026, 0.0008],
…,
[-0.0003, -0.0022, -0.0003, …, 0.0001, 0.0012, -0.0007],
[ 0.0033, 0.0045, 0.0018, …, -0.0020, -0.0036, 0.0007],
[ 0.0031, 0.0045, 0.0015, …, -0.0022, -0.0012, -0.0007]],
requires_grad=True)

After dropout:

tensor([[ 0.0000, 0.0059, 0.0006, …, -0.0037, -0.0000, -0.0030],
[ 0.0011, 0.0021, 0.0030, …, 0.0057, 0.0000, 0.0021],
[ 0.0000, 0.0000, 0.0030, …, -0.0070, -0.0038, 0.0000],
…,
[-0.0000, -0.0032, -0.0004, …, 0.0000, 0.0000, -0.0010],
[ 0.0047, 0.0065, 0.0026, …, -0.0029, -0.0000, 0.0010],
[ 0.0000, 0.0065, 0.0021, …, -0.0031, -0.0018, -0.0009]])

Why does it happen?

2

Hello csailnadi,

Take a look at the documentation for Dropout: https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout

To keep the same output scale (in expectation), outputs are rescaled by 1/(1-p) during training. You can see this in your example: for instance 0.0041 is mapped to 0.0041 / 0.7 = 0,00585 ~ 0.0059.

Hope this helped :slight_smile:

1 Like