Filter normalisation

(Sebastienwood) #1

Hi :slight_smile:

I’m trying to multiply filters by the average value of another filter of the same dimension (that is a copy of this filter, with few modifications). When I get params they look like 4 dimensional or 3 dimensional tensors, with if I got it correctly the filter are at the last two dims.

Then I do :
param_tensor *= torch.mean(param_tensor, (2,3)).expand_as(param_tensor)

I expect that the tensor would thus have an element wise mult with the average being expanded in the 2 lasts dimensions.

However it fails with the following error message : The expanded size of the tensor (3) must match the existing size (16) at non-singleton dimension 2. Target sizes: [16, 3, 3, 3]. Tensor sizes: [16, 3].

I looked into weight norm but it seems slightly different from what I want to achieve. If anyone knows how to do so it would help, thanks ! :slight_smile:

#2

Try to pass keepdim=True to the mean call:

param_tensor *= torch.mean(param_tensor, (2,3), keepdim=True).expand_as(param_tensor)