# Clarity on default initialization in pytorch

**URL:** <https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696>\
**Category:** Uncategorized\
**Created:** [June 9, 2020, 12:02am UTC](https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696 "2020-06-09T00:02:05Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![Nikronic](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/32/71873_2.png) [@Nikronic](https://discuss.pytorch.org/u/Nikronic)\
**Post date:** [June 9, 2020, 1:30pm UTC](https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696/2 "2020-06-09T13:30:35Z")

</div>

Hi,

Let me explain it step by step.

1. Here is [kaiming\_uniform\_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_).  

2. In [linear implementation](https://pytorch.org/docs/stable/nn.html#torch.nn.Linear) code you referenced:  
 ![image](https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2d67360b59665908a9ee8b4916e926f6ad7e97d.png)  

So what we have here is that `k= 1/in_feautres` which in case of kaming it can be represented `k=1/fan_in`. Also, we want a boundary of `[-sqrt(k), sqrt(k)]` where `k = bound^2= 1 / fan_in` from step 1.

For simplcity, just replace `sqrt(5)` in `gain` formula then optain `bound` in `kaiming_uniform_` and replace the `bound` as `k` in linear.

Edit: Add some related posts

1. [Kaiming init of conv and linear layers, why gain = sqrt(5) · Issue #15314 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/15314)
2. [Why the default negative\_slope for kaiming\_uniform initialization of Convolution and Linear layers is √5?](https://discuss.pytorch.org/t/why-the-default-negative-slope-for-kaiming-uniform-initialization-of-convolution-and-linear-layers-is-5/29290)

Bests

---

_[View the full topic](https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696)._