Randomly set the weights of part of model layers for every time training

  • Is there some good way to set(re-set ) part layers’ weights randomly for every time train the model? (‘every time’ means when every time train one time epoch.)

  • In addition, What is the usual range of weights?

What do you mean by setting and resetting weights at different layers??

You can look at different initialisation like Xavier glorot initialisation and some other.
Initalisation of weights depend on what data you are working with.

I’m sorry, I did not express clearly.
I mean, for specific layers of a model, set(re-set) layers’ weights randomly for every time training. Is there some good way to do this?

I did not get your question well. But to my understanding.
If building your own function like convolution or something == True
you can initialise your weights as torch.nn.Parameter.

else
you can use smthing like this:

def init_params(m):
  if type(m)==nn.Linear or type(m)==nn.Conv2d:
    m.weight.data=torch.randn(m.weight.size())*.01#Random weight initialisation
    m.bias.data=torch.zeros(m.bias.size())

1 Like

Appreciate your timely help.:blush:
What does the ‘m’ in the function mean as a parameter?
I guess it’s each layer in the model.

  • If so, which function in PyTorch I should use to get the layers?
  • If not, would you please tell me what it means?

m in my previous reply is a layer.

a simple example of layer or say a for example a 2D-Convolutional layer maybe:

import torch.nn as nn
layer=nn.Conv2d(in_channels=1,out_channels=32,kernel_size=3,stride=1)

torch.nn.Conv2D

I think that is too basic a question. Never mind but. You can check out Deep learning with pytorch Courses online.One such course

I’m sorry, again, I didn’t express clearly.:confused:
I mean, if the ‘m’ is a layer, then how to get and re-set the layer’s weights during the period of training the model? is there some good way to do this?

def init_params(m):
  if type(m)==nn.Linear or type(m)==nn.Conv2d:
    m.weight.data=torch.randn(m.weight.size())*.01#Random weight initialisation
    m.bias.data=torch.zeros(m.bias.size())

#for setting the weights you can use:
Model.apply(init_params) #Model here is the model that you have created

1 Like

It’s the first time I see the use of function, model.apply(). It’s very fantastic.
Thank you very much for your help :grinning:

1 Like