# Weight initilzation **URL:** \ **Category:** Uncategorized\ **Created:** [January 23, 2017, 11:18pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157 "2017-01-23T23:18:53Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![mratsim](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mratsim/32/548_2.png) [@mratsim](https://discuss.pytorch.org/u/mratsim)\ **Post date:** [May 29, 2017, 8:16am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/42 "2017-05-29T08:16:59Z")
@lakehanne Iirc, Python allows something like 256 recursive calls, either you forgot a finishing condition in your code or you need another approach (iterator/generator?) ---
**Author:** ![Xiaoyu\_Liu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xiaoyu_liu/32/285_2.png) [@Xiaoyu\_Liu](https://discuss.pytorch.org/u/Xiaoyu_Liu)\ **Post date:** [May 30, 2017, 5:28pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/43 "2017-05-30T17:28:18Z")
@Atcold Hi, can we simply use model2’s weights to initialize model1’s layer by `model1.conv1.parameters = model2.conv1.parameters` ? Thank you. ---
**Author:** ![Vincent\_Zhang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vincent_zhang/32/864_2.png) [@Vincent\_Zhang](https://discuss.pytorch.org/u/Vincent_Zhang)\ **Post date:** [June 27, 2017, 12:20pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/44 "2017-06-27T12:20:15Z")
@Xiaoyu_Liu You can check[this](https://discuss.pytorch.org/t/are-there-any-recommended-methods-to-clone-a-model/483/6?u=vincent_zhang) for example. Further more, if you want to copy by module names, you can choose `named_parameters()` or `state_dict()`, and make sure of the deep copy. ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [July 19, 2017, 11:20pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/45 "2017-07-19T23:20:22Z")
where is the implementation of Xavier? ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [July 19, 2017, 11:28pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/46 "2017-07-19T23:28:10Z")
is the xavier: ``` m.weight.data.normal_(0, math.sqrt(2. / n)) ``` ??? ---
**Author:** ![vabh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/898d66/32.png) [@vabh](https://discuss.pytorch.org/u/vabh)\ **Post date:** [July 25, 2017, 8:34am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/47 "2017-07-25T08:34:07Z")
Here you go: > ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [August 4, 2017, 7:19pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/48 "2017-08-04T19:19:56Z")
how did u know that apply existed? doesn’t seem to be documented wherever Im looking. ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [August 7, 2017, 7:38pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/49 "2017-08-07T19:38:34Z")
I guess one can find the official function in the docs: [http://pytorch.org/docs/master/nn.html?highlight=xavier#torch.nn.init.xavier\_normal](http://pytorch.org/docs/master/nn.html?highlight=xavier#torch.nn.init.xavier_normal) ---
**Author:** ![lantiga](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lantiga/32/5523_2.png) [@lantiga](https://discuss.pytorch.org/u/lantiga)\ **Post date:** [August 7, 2017, 10:18pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/50 "2017-08-07T22:18:26Z")
You’re right, `apply` is not documented. I’ll open a PR. Updated: [https://github.com/pytorch/pytorch/pull/2327](https://github.com/pytorch/pytorch/pull/2327) ---
**Author:** ![diggerdu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diggerdu/32/1457_2.png) [@diggerdu](https://discuss.pytorch.org/u/diggerdu)\ **Post date:** [September 2, 2017, 6:55am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/51 "2017-09-02T06:55:19Z")
module.weight.data.copy\_(everythings\_you\_want) conv2d weights [Channels, Groups, Height, Width] ---
**Author:** ![will\_soon](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/f08c70/32.png) [@will\_soon](https://discuss.pytorch.org/u/will_soon)\ **Post date:** [October 21, 2017, 2:01am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/52 "2017-10-21T02:01:09Z")
It seems that there are only conv2 layer’s initilzation , not linear layer’s in resnet model. How about linear layer? Is there any default initilzation if I don’t define the initilzation in my model? Thank you! ---
**Author:** ![litesaber](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/litesaber/32/2360_2.png) [@litesaber](https://discuss.pytorch.org/u/litesaber)\ **Post date:** [November 19, 2017, 10:20pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/53 "2017-11-19T22:20:29Z")
Looks like glorot uniform is the default initialization for nn.Linear. Check out [`reset_parameters()`](http://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear). ---
**Author:** ![yangn1](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/bb73d2/32.png) [@yangn1](https://discuss.pytorch.org/u/yangn1)\ **Post date:** [November 28, 2017, 1:49pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/54 "2017-11-28T13:49:35Z")
Code god! My I ask you question—\>How can I INIT a LSTM or CNN. I have known that A LSTMCell can be INTIed as \ OR \ . BUT LSTM can’t…that. ---
**Author:** ![weidler](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weidler/32/26047_2.png) [@weidler](https://discuss.pytorch.org/u/weidler)\ **Post date:** [November 29, 2017, 8:54pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/55 "2017-11-29T20:54:44Z")
according to dir(net) there are two weight matrices: ``` weight_hh_l0 weight_ih_l0 ``` so you could do ``` def initialize_weights(model): if type(model) in [nn.Linear]: nn.init.xavier_normal(model.weight.data) elif type(model) in [nn.LSTM, nn.RNN, nn.GRU]: nn.init.xavier_normal(model.weight_hh_l0) nn.init.xavier_normal(model.weight_ih_l0) ``` This is also documented [here](http://pytorch.org/docs/master/nn.html?highlight=xavier#torch.nn.RNN). So if you have more than 1 recurrent layer, you’ll have to initialize _2 \* recurrent\_layers_ weight matrices. Not sure how to elegantly iterate over them, using _list(parameters())_ and only changing 2 dimensional tensors (i.e. not biases) could be a solution. ---
**Author:** ![yangn1](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/bb73d2/32.png) [@yangn1](https://discuss.pytorch.org/u/yangn1)\ **Post date:** [November 30, 2017, 1:57am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/56 "2017-11-30T01:57:30Z")
how about multi-Layer? I can Init one Layer yet, but I cant Inti multi-layer by using a “loop” , too. And I have tried some way， but it doesn’t make sense… ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [December 1, 2017, 9:53am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/57 "2017-12-01T09:53:52Z")
You can call the `weight_init` method on your model using `model.apply(weight_init)`. ---
**Author:** ![Neeharika\_Noni](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neeharika_noni/32/3514_2.png) [@Neeharika\_Noni](https://discuss.pytorch.org/u/Neeharika_Noni)\ **Post date:** [March 20, 2018, 3:46pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/58 "2018-03-20T15:46:50Z")
Does batch normalization help in different weight initializations.ensure you include at least one of Xavier’s and Kaiming’s initializations.can any one please help me for this ---
**Author:** ![ForeverZH0204](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/foreverzh0204/32/9493_2.png) [@ForeverZH0204](https://discuss.pytorch.org/u/ForeverZH0204)\ **Post date:** [March 29, 2018, 1:08pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/59 "2018-03-29T13:08:47Z")
Hi I read these answer just now.I have a question that: With ``` a = nn.Linear(2,3) ``` I got a fc layer with 2d input and 3d output, and the params in the layer are from a N(0, 1) without my initialization , i.e. the fc layer is initialized automatically. Can i specify the initialization method like xavier when i initial the layer,or i have to got the params iteratelly and use a function to change it or with the help of apply method? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 29, 2018, 1:20pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/60 "2018-03-29T13:20:39Z")
The `Linear` layer will return a 2 dimensional output with 3 output features. It’s initialized automatically using the `uniform_` distribution. You can see the initialization [here](https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L48). You could override the layer and implement your method or just use `apply` to init it with xavier or another method. ---
**Author:** ![icjwk](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/e9a140/32.png) [@icjwk](https://discuss.pytorch.org/u/icjwk)\ **Post date:** [December 1, 2018, 5:20pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/62 "2018-12-01T17:20:52Z")
no such thing as should, ask any is ok [Previous page](https://discuss.pytorch.org/t/weight-initilzation/157.md?page=2) [Next page](https://discuss.pytorch.org/t/weight-initilzation/157.md?page=4)