How are layer weights and biases initialized by default?

42

Is it a good idea to initialize weights itself in the init function by looping over the layers?

Class Net(nn.Module):
        def  __init__():
            #...... Defining Layers
            #.......................

            for m in self.modules():
                 if isinstance(m, nn.Conv3d):
                      m.weight = nn.init.kaiming_normal(m.weight, mode='fan_out')
                 elif isinstance(m, nn.BatchNorm3d) or isinstance(m, nn.BatchNorm2d):
                      m.weight.data.fill_(1)
                      m.bias.data.zero_()
        
        def forward():
43

Yes, this should be working, if all modules were already initialized.

44

You can also use as this function:

def initialize_parameters(m):
    if isinstance(m, nn.Conv2d):
        nn.init.kaiming_normal_(m.weight.data, nonlinearity = 'relu')
        nn.init.constant_(m.bias.data, 0)
    elif isinstance(m, nn.Linear):
        nn.init.xavier_normal_(m.weight.data, gain = nn.init.calculate_gain('relu'))
        nn.init.constant_(m.bias.data, 0)
45

Could you clarify why we donā€™t use .data attribute anymore with nn.init? The documentation for the init functions refer to input Tensors and not Parameters still.

(Also, as an aside: I lurk on the Pytorch forums a lot, thank you for all your extremely helpful responses.)

46

Alban explains it here nicely.

47

I though it was init.kaiming_uniform as per line 87 here?

48

Kaiming uniform would initialise with variance 2 / fan_in.
However, with a=math.sqrt(5), the initialisation ends up with a variance 1 / (3 * fan_in), which does not correspond to any standard initialisation scheme.

I refer to my earlier reply for more detailsā€¦

49

Hi Ptrblck,

I hope you are well. Sorry, I wan to train my classifier with 10 ensembles. The different between ensembles are the order of the subjects that I used for creating data. I want to be sure that for each ensemble the weight initialization is different from other. ensembles are running in parallel jobs independent from each other.

I just define my model and use it directly . can I be sure that for each ensemble the weight initialization is different ? for example randomly be different.

50

Yes, as long as you donā€™t set the random seed before initializing each module, the parameters would be different. You can check (some or all) via print(modelX.my_layer.param) where the X denotes the current model and would see that the same parameter would have different values after initializing all models.

1 Like
51

many thanks. I really appreciate your help

52

What difference would it make if we donā€™t set nonlinearity=ā€˜reluā€™ while the nonlinearity is layer is relu, because the default nonlinearity is leaky relu

53

The calculate_gain is using the specified nonlinearity as seen here:

Return the recommended gain value for the given nonlinearity function.
    The values are as follows:
    ================= ====================================================
    nonlinearity      gain
    ================= ====================================================
    Linear / Identity :math:`1`
    Conv{1,2,3}D      :math:`1`
    Sigmoid           :math:`1`
    Tanh              :math:`\frac{5}{3}`
    ReLU              :math:`\sqrt{2}`
    Leaky Relu        :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}`
    SELU              :math:`\frac{3}{4}`
    ================= ====================================================

which is then used to compute std as seen here.

1 Like
54

[quote=ā€œRushirajsinh_Parmar, post:44, topic:13073ā€]

isinstance(m, nn.Linear):
        nn.init.xavier_normal_(m.weight.data, gain = nn.init.calculate_gain('relu'))
        nn.init.constant_(m.bias.data, 0)

[/quote] HI. Assume that using xavier in linear layers isnt good idea. Have you got good results with this init? Because during this from init.py

x = torch.randn(512)
import math
std = math.sqrt(2.0 / float(512 + 512))
a = math.sqrt(3.0) * std

for i in range(100):
y = torch.Tensor(512,512).uniform_(-a,a)
x = torch.relu(x@y)

print(x.mean(),x.std())
tensor(8.1831e-16) tensor(1.0922e-15)

this tensors are very small for calculating grad

1 Like
55

Hi, Iā€™m not sure about the custom initialization, I havenā€™t tried this yet, Iā€™ve been sticking to xavier initialisation for almost all the applications but will give this a try to see how it compares with xavier!

1 Like
56

Hey :wave: . this is implementation of xavier init from source code of pytorch and also i read that xavier is better if use symmetric functions like sigm or tanh and cause of that i experiment like that (with relu) and get confidence that using xaiver without sigm or tanh is not good approach. share recently received learning with you

57

if you had better result with xavier init where you had non symmetric function instead of without xavier , let me know i will be glad. because if i have apply sigmoid function in my little code that return me more reliable gradients. also you can check it yourself)))

59

Your encoder_net works fine for inputs in the shape [batch_size, 3, 32, 32]:

encoder_net = nn.Sequential(
              nn.Conv2d(3, 64, 4, stride=2, padding=0),
              nn.ReLU(),
              nn.Conv2d(64, 128, 4, stride=2, padding=0),
              nn.ReLU(),
              nn.Conv2d(128, 512, 4, stride=2, padding=0),
              nn.ReLU(),
              # nn.Conv2d(256, 512, 2, stride=2, padding=0),
              # nn.ReLU(),
              nn.Flatten(),
              nn.Linear(2048,32)  #512*4
)

x = torch.randn(16, 3, 32, 32)
out = encoder_net(x)
print(out.shape)
# torch.Size([16, 32])
63

Iā€™m unsure where the error is coming from, but could you try to slim down the code a bit and check which model fails with which input shapes?