# What's the default initialization methods for layers?

**URL:** <https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157>\
**Category:** Uncategorized\
**Created:** [May 17, 2017, 1:40pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157 "2017-05-17T13:40:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nick\_Young](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nick_young/32/599_2.png) [@Nick\_Young](https://discuss.pytorch.org/u/Nick_Young)\
**Post date:** [May 17, 2017, 1:40pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/1 "2017-05-17T13:40:33Z")

</div>

what’s the default initialization methods for layers? Like conv, fc, and RNN layers? are they just initialized to all zeros?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [May 17, 2017, 2:52pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/2 "2017-05-17T14:52:11Z")

</div>

All the layer are implemented in this folder: [https://github.com/pytorch/pytorch/tree/master/torch/nn/modules](https://github.com/pytorch/pytorch/tree/master/torch/nn/modules)  
The initialization depend on the layer, for example, the linear one is [here](https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L44-L48)

---

<div class="post-metadata">

**Author:** ![Nick\_Young](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nick_young/32/599_2.png) [@Nick\_Young](https://discuss.pytorch.org/u/Nick_Young)\
**Post date:** [May 17, 2017, 3:16pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/3 "2017-05-17T15:16:36Z")

</div>

Thank you so much!👍

---

<div class="post-metadata">

**Author:** ![Konpat\_Ta\_Preechakul](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/konpat_ta_preechakul/32/3311_2.png) [@Konpat\_Ta\_Preechakul](https://discuss.pytorch.org/u/Konpat_Ta_Preechakul)\
**Post date:** [February 3, 2018, 5:37am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/4 "2018-02-03T05:37:02Z")

</div>

I see only `torch.Tensor(...)` without any specific initialization method. I wonder what would that be?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 3, 2018, 2:43pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/5 "2018-02-03T14:43:59Z")

</div>

In `reset_parameters()` the weights are set/reset.

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 7, 2018, 1:18am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/6 "2018-07-07T01:18:25Z")

</div>

Are these initialization basically He init or Xavier init?

---

<div class="post-metadata">

**Author:** ![klory](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/klory/32/5814_2.png) [@klory](https://discuss.pytorch.org/u/klory)\
**Post date:** [July 11, 2018, 11:10am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/7 "2018-07-11T11:10:32Z")

</div>

Tanh --\> Xavier  
ReLU --\> He

> <https://stats.stackexchange.com/questions/319323/whats-the-difference-between-variance-scaling-initializer-and-xavier-initialize/319849#319849>

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 12:50am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/8 "2018-07-14T00:50:50Z")

</div>

So Pytorch uses He when its ReLU? Im confused what pytorch does.

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 1:23am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/9 "2018-07-14T01:23:42Z")

</div>

Sorry ptrblck, Im confused…pytorch uses Xavier or He depending on the activation? Thats what klory seems to imply but the code looks as follows:

```
def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)
```

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [July 14, 2018, 2:01am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/10 "2018-07-14T02:01:35Z")

</div>

> Tanh → Xavier  
> ReLU → He

No that’s not correct, PyTorch’s initialization is based on the layer type, not the activation function (the layer doesn’t know about the activation upon weight initialization).

For the linear layer, this would be somewhat similar to He initialization, but not quite:

> <https://github.com/pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52>

```
def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)

```

Ie., when I remember correctly, He init is “sqrt(6 / fan\_in)” whereas in PyTorch Linear layer it’s “1. / sqrt(fan\_in)”

---

<div class="post-metadata">

**Author:** ![klory](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/klory/32/5814_2.png) [@klory](https://discuss.pytorch.org/u/klory)\
**Post date:** [July 14, 2018, 2:10am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/11 "2018-07-14T02:10:51Z")

</div>

Yeah, you’re correct, I just check their code for linear.py and conv.py, it seems they’re all using Xavier right? (I got the Xavier explanation from [here](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization))

> <https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/linear.py#L48-L52>

> <https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/conv.py#L40-L47>

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 2:11am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/12 "2018-07-14T02:11:10Z")

</div>

are they all using Xavier?

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [July 14, 2018, 2:41am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/13 "2018-07-14T02:41:38Z")

</div>

> it seems they’re all using Xavier right?

Doesn’t xavier also include fan\_out though? Here, I can only see input channels, not output channels.

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 2:41am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/14 "2018-07-14T02:41:59Z")

</div>

so is it just a unpublished made up pytorch init?

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [July 14, 2018, 3:08am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/15 "2018-07-14T03:08:02Z")

</div>

Maybe I am overlooking sth or don’t recognize it, but I think so

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 3:10am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/16 "2018-07-14T03:10:11Z")

</div>

A validation from someone in the pytorch team would be nice calling for call for master @SimonW 😉 🙂

---

<div class="post-metadata">

**Author:** ![SimonW](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/32/1702_2.png) [@SimonW](https://discuss.pytorch.org/u/SimonW)\
**Post date:** [July 14, 2018, 10:40am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/17 "2018-07-14T10:40:39Z")

</div>

no, they are from respective well-established published papers actually. e.g., linear init is from “Efficient Backprop”, LeCun’99.

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 10:24pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/18 "2018-07-14T22:24:41Z")

</div>

Thanks! I appreciate the it.

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [July 14, 2018, 10:25pm UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/19 "2018-07-14T22:25:19Z")

</div>

> [@SimonW](#):
>
> “Efficient Backprop”, LeCun’99

this one I assume:

> **[lecun-98b.pdf](https://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)**

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [July 15, 2018, 12:11am UTC](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/20 "2018-07-15T00:11:34Z")

</div>

Maybe it would be worthwhile adding comments to the docstrings? Would make it easier for the next person to find plus more convenient to refer to in papers.

[Next page](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157.md?page=2)