Using kaiming initialization

1

For my current neural network I’d like to initialize my model’s weights with the torch.nn.init.kaiming_normal, so I have tried this :
torch.nn.init.kaiming_normal(m.weight.data)
but I receive this error message :
Fan in and fan out can not be computed for tensor with less than 2 dimensions
Is the less than a strictly less than ? I am not sure I understand why it’s failing, if someone could give me some pointers that would be great.

1 Like
2

If you put print(m.weight.data.size() before the kaiming init, what does it show?

3

It shows this :

torch.Size([200, 1000])
torch.Size([200, 400])
torch.Size([100, 200])
torch.Size([2, 100])
torch.Size([1000])

I’m guessing the last one is the one causing problems then because it’s one dimensional ?

4

What type of layer is that last one from?
I am guessing that it is a layer of a type that you can’t use kaiming init on.

5

It corresponds to a BatchNorm layer, so I am guessing you are right about the kaiming initialization not being possible on that. I should have thought about it ^^. Thank you for your help with that!

2 Likes