Translate weight initialization from tensorflow to pytroch

[Mikel_B]

How would you go about initializing this TF fully connected layer

Dense(2, activation=‘linear’, kernel_initializer=RandomNormal(mean=0.0, stddev=0.2, seed=seed),
bias_initializer=Constant(value=[3.,-3.]), name=‘pi’)

in PyTorch?

I tried this:

self.pi = nn.Linear(input_dim, 2)
nn.init.normal_(self.pi.weight, mean=0, std=0.2)
nn.init.constant_(self.pi, 0)

But I am not sure about the last line…
Thank you for your time

[Nikronic (Nikan Doosti)]

Hi,

About the last line, you need to initialize it like the way you have initialized your weights. In other words, use self.pi.bias instead of self.pi which is the module.

Also, nn.init.constant_(self.pi, 0) initialized the entire bias tensor with zeroes while the TF counterpart has initialized it with [3, -3]. To do so, you can simply assign a tensor with these values to the bias attribute of self.pi:

with torch.no_grad():
    self.pi.bias = nn.Parameter(torch.tensor([3., -3.]))

How to manually set the weights in a two layer linear model?

You would just need to wrap it in a torch.no_grad() block and manipulate the parameters as you want: model = torch.nn.Sequential(nn.Linear(10, 1, bias=False)) with torch.no_grad(): model[0].weight = nn.Parameter(torch.ones_like(model[0].weight)) model[0].weight[0, 0] = 2. model[0].weight.fill_(3.) ...