What is the theoretical reason for the default way of Pytorchs weight initialization?

[Nikronic (Nikan Doosti)]

Hi,

I think this thread and its references can answer your question.

Clarity on default initialization in pytorch

According to the documentation for torch.nn, the default initialization uses a uniform distribution bounded by 1/sqrt(in_features), but this code appears to show the default initialization as Kaiming uniform. Am I correct in thinking these are not the same thing? And if so, perhaps the documentation can be updated? Does anyone know the motivation for this choice of default? In particular, the code appears to define a negative slope for the subsequent nonlinearity of sqrt(5), whereas the default…

Feel free to ask if it did not help.

Bests