# Weight initilzation

**URL:** <https://discuss.pytorch.org/t/weight-initilzation/157>\
**Category:** Uncategorized\
**Created:** [January 23, 2017, 11:18pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157 "2017-01-23T23:18:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 23, 2017, 11:18pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/1 "2017-01-23T23:18:53Z")

</div>

How I can initialize variables, say Xavier initialization?

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 23, 2017, 11:32pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/2 "2017-01-23T23:32:01Z")

</div>

Hi @Hamid,

I think you can extract the network’s parameters `params = list(net.parameters())` and then apply the initialisation you may like.  
If you need to apply the initialisation to a specific `module`, say `conv1`, you can extract the specific parameters with `conv1Params = list(net.conv1.parameters())`. You will have the kernels in `conv1Params[0]` and the bias terms in `conv1Params[1]`.

---

<div class="post-metadata">

**Author:** ![fmassa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/32/58_2.png) [@fmassa](https://discuss.pytorch.org/u/fmassa)\
**Post date:** [January 23, 2017, 11:33pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/3 "2017-01-23T23:33:17Z")

</div>

Another possibility, which is present in the examples, can be found [here](https://github.com/pytorch/examples/blob/master/dcgan/main.py#L90-L96). This function specifies how the weights should be handled, and the weights are modified in [this line](https://github.com/pytorch/examples/blob/master/dcgan/main.py#L131).

---

<div class="post-metadata">

**Author:** ![michael\_k](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_k/32/64_2.png) [@michael\_k](https://discuss.pytorch.org/u/michael_k)\
**Post date:** [January 24, 2017, 12:43am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/4 "2017-01-24T00:43:08Z")

</div>

[Another initialization example](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L112-L118) from PyTorch Vision resnet implementation.

---

<div class="post-metadata">

**Author:** ![Kalamaya](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kalamaya/32/381_2.png) [@Kalamaya](https://discuss.pytorch.org/u/Kalamaya)\
**Post date:** [January 24, 2017, 2:15am UTC](https://discuss.pytorch.org/t/weight-initilzation/157/5 "2017-01-24T02:15:54Z")

</div>

@Atcold Can you give an example of what you mean? Thanks!

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 4:14pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/6 "2017-01-24T16:14:27Z")

</div>

@Kalamaya, I believe @fmassa is a cleaner solution.  
You traverse all `Module`s, and, upon ` __class__. __name__ ` matching, you initialise the parameters with what you prefer.  
My method presupposes that you know the order of the `Module`s in the `_modules` `OrderedDict()`.

Does it make any sense what I am saying? If it does not, I can try better and with an example.

---

<div class="post-metadata">

**Author:** ![Kalamaya](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kalamaya/32/381_2.png) [@Kalamaya](https://discuss.pytorch.org/u/Kalamaya)\
**Post date:** [January 24, 2017, 4:18pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/7 "2017-01-24T16:18:38Z")

</div>

@Atcold Yes, an example will help here since I am still traversing unfamiliar territory… 🙂 thank you for your help in advanced! Much obliged. If it helps, I am basically trying to initialize my conv and fully connected layers that I have. (I’d like to do Xavier, or fan\_in fan\_out etc). Thanks!

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 4:34pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/8 "2017-01-24T16:34:45Z")

</div>

You first define your name check function, which applies selectively the initialisation.

```python
def weights_init(m):
    classname = m. __class__. __name__
    if classname.find('Conv') != -1:
        xavier(m.weight.data)
        xavier(m.bias.data)

```

Then you traverse the whole set of `Modules`.

```python
net = Net() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init

```

And this is it. You just need to define the `xavier()` function.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [January 24, 2017, 4:47pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/9 "2017-01-24T16:47:14Z")

</div>

A less Lua way of doing that would be to check if some module is an instance of a class. This is the recommended way:

```python
def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)

```

@Atcold another thing, accessing members prefixed with underscore is not recommended. They’re internal and subject to change without notice. If you want to get a iterator over modules use `.modules()` (searches recusrively) or `.children()` (only one level).

---

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 24, 2017, 7:37pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/10 "2017-01-24T19:37:04Z")

</div>

Thanks guys.  
I am trying to apply weight initialization to a fully connected network (nn.Linear). I need, however, fan\_out and fan\_in of this layer. By fan\_out and fan\_in I mean number of output neurons and input neurons, respectively. How can I access them?

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 7:51pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/11 "2017-01-24T19:51:52Z")

</div>

@Hamid, you can check the size of the `weight` matrix.

```python
size = m.weight.size() # returns a tuple
fan_out = size[0] # number of rows
fan_in = size[1] # number of columns

```

@apaszke, thanks for the heads-up! I’m still new to the Python world…

Edit: apply @apaszke fix.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [January 24, 2017, 7:54pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/12 "2017-01-24T19:54:44Z")

</div>

A small note - `.size()` is also defined on Variables, so no need to unpack the data. `m.weight.size()` will work too.

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 8:37pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/14 "2017-01-24T20:37:04Z")

</div>

@Hamid, are you trying to ask something? I am not sure I understand.  
Could you also please format your code with three backticks and the word `python`, so that I can read what you posted?

---

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 24, 2017, 8:40pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/15 "2017-01-24T20:40:59Z")

</div>

@Atcold, by checking if isinstance(m, nn.Linear) it would apply to linear module, correct?  
If I call the weight initialization, it would be applied to all layers?  
I have a residual module each having 2 linear layers. Then several of these modules.

my code:

```
def weight_init(m): 
	if isinstance(m, nn.Linear):
		size = m.weight.size()
		fan_out = size[0] # number of rows
		fan_in = size[1] # number of columns
		variance = np.sqrt(2.0/(fan_in + fan_out))
		m.weight.data.normal_(0.0, variance)

class Residual(nn.Module):
	def __init__ (self,dropout, shape, negative_slope, BNflag = False):
		super(Residual, self). __init__ ()
		self.dropout = dropout
		self.linear1 = nn.Linear(shape[0],shape[1])
		self.linear2 = nn.Linear(shape[1],shape[0])
		self.dropout = nn.Dropout(self.dropout)
		self.BNflag = BNflag
		self.batch_normlization = nn.BatchNorm1d(shape[0])
		self.leakyRelu = nn.LeakyReLU(negative_slope = negative_slope , inplace=False)

	def forward(self, X):
		x = X
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear1(x)
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear2(x)
		x = torch.add(x,X)
		return x
		
		
class FullyCN(nn.Module):
	def __init__ (self, args):
		super(FullyCN, self). __init__ ()
		self.numlayers = arg.sm-num-hidden-layers
		self.learning-rate= args.sm-learning-rate
		self.dropout = arg.sm-dropout-prob
		self.BNflag = args.sm-bn
		self.shape = [args.sm-input-size,args.sm-num-hidden-units]		
		self.res = Residual(self.dropout,self.shape,args.sm_act_param,self.self.BNflag)
		self.res(weight_init)
		self.res-outpus = []

	def forward(self,X):
		self.res-outpus.append(self.res(X))
		for i in range(self.numlayers):
			self.res-outpus.append(self.res(self.res-outpus[-1]))
		return self.res-outpus[-1]
```

---

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 24, 2017, 8:43pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/16 "2017-01-24T20:43:37Z")

</div>

sorry about confusion

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 8:45pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/17 "2017-01-24T20:45:21Z")

</div>

> [@Hamid](#):
>
> @Atcold, by checking if isinstance(m, nn.Linear) it would apply to linear module, correct?

Correct.

> [@](#):
>
> If I call the weight initialization, it would be applied to all layers?

Yup. All **linear** layers.

> [@](#):
>
> I have a residual module each having 2 linear layers. Then several of these modules.

Sure, it will apply the normalisation to each `Module` that belongs to the class `nn.Linear`.

---

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 24, 2017, 8:48pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/18 "2017-01-24T20:48:45Z")

</div>

> [@Hamid](#):
>
> self.res(weight\_init)

But I have called weight\_init once for the class while I call linear layers in a for loop (i.e., there are multiple sets of variables).

---

<div class="post-metadata">

**Author:** ![Atcold](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atcold/32/47260_2.png) [@Atcold](https://discuss.pytorch.org/u/Atcold)\
**Post date:** [January 24, 2017, 8:53pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/19 "2017-01-24T20:53:26Z")

</div>

```python
net = Residual() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init

```

I’m not too sure what you’re doing with `FullyCN`…

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [January 24, 2017, 8:54pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/20 "2017-01-24T20:54:15Z")

</div>

The `apply` function will search recursively for all the modules inside your network, and will call the function on each of them. So all `Linear` layers you have in your model will be initialized using this one call.

---

<div class="post-metadata">

**Author:** ![Hamid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/32.png) [@Hamid](https://discuss.pytorch.org/u/Hamid)\
**Post date:** [January 24, 2017, 9:07pm UTC](https://discuss.pytorch.org/t/weight-initilzation/157/21 "2017-01-24T21:07:00Z")

</div>

@Atcold, in FullyCN I use several residual modules. In another piece of code, I pass data to FullyCN which returns corresponding output via its forward function.

[Next page](https://discuss.pytorch.org/t/weight-initilzation/157.md?page=2)