What are the default initialised matrix values of the convolution kernels in PyTorch?

[CCL]

Different kernel values are useful for different tasks, e.g. edge detection. What kernel matrix values does PyTorch use by default?

[Nikronic (Nikan Doosti)]

This might help

Clarity on default initialization in pytorch

According to the documentation for torch.nn, the default initialization uses a uniform distribution bounded by 1/sqrt(in_features), but this code appears to show the default initialization as Kaiming uniform. Am I correct in thinking these are not the same thing? And if so, perhaps the documentation can be updated? Does anyone know the motivation for this choice of default? In particular, the code appears to define a negative slope for the subsequent nonlinearity of sqrt(5), whereas the default…

Bests

[dkleine (Daniel Kleine)]

And if so, perhaps the documentation can be updated? Does anyone know the motivation for this choice of default?

I absolutely agree with that statement in the linked post. It took me quite a lot of time to find a comprehensive answer to the question “what are the default values for the kernel used in a convolutional layer in a CNN?”.

I think the documentation needs some improvements here. Some suggestions:

• default initializations (torch.nn.init) need to be linked to relevant pages, e.g. Conv2D
• the web page to default initializations (torch.nn.init) needs more documentation, e.g. that “Conv{1,2,3}D” refers to values (weights) used in the convolution kernel initialized by default. There is also no information that it uses Kaiming uniform distribution (for me as a beginner in CV, this was not that obvious).

[ptrblck]

These sound like great suggestions! Would you be interesting in adding these to the documentation?

[dkleine (Daniel Kleine)]

Sure, I would be happy to contribute!