PyTorch weight initialization

1

Hi all,

Can I ask you something about weight initialization in PyTorch because I have been confused?

First of all, is it mandatory to initialize weights in PyTorch or can the framework initialize the layers for us?

Also, because many posts are outdated, in default initialization what kind of initialization the framework use? Xavier?

Moreover, all layers provide default initialization (LSTMS, GRUs, CNNs layers etc.)

Finally, if I use init function to ‘manually’ initialize the layers. Where I must put this function? inside the class where I define the layers?

2

Hi,

All of our layers use a default initialization that is the most commonly used in the literature. You can see in the doc what is the default initialization.
If you want to change the default initialization, you can use the nn.init module’s functions to change the weights of your modules after creating them. There is no hard rule how to do this. You can simply update the .weight after you create it. Or do whatever feels best for your use case.