Weight initialization in Pytorch?

vision
#1

I am new to PyTorch and machine learning in general and I wanted to make sure I understand weight initialization correctly. as I understand weight initialization can be done to all layers but if we choose to do it ourselves, we use the nn.init.any_type_of_weight_initialization(moudle.weight, *args). so I understand that the purpose is to replace the already initialized default weights(by module.weight) by the any_type_of_weight_initialization method we want. but while trying it, I found both module.weight and the one we initialized have the same weight entries. so my questions are:

  1. what is the default weight initialization method used in module.weight?
  2. do we really replace the already initialized weights by other methods or I am misunderstanding the whole process? thanks in advance
#2

  1. The default initialization depends on the used nn.Module and can be found in its reset_parameters() method. E.g. here for nn.Linear.

  2. You are understanding the workflow correctly. The model initialization via model = MyModel() will initialize all parameters using their default method (unless you’ve already manipulated the MyModel.__init__ to use a custom parameter initialization). Afterwards you are usually calling into a custom weight_init method via model.apply().