Initializing weights and adding trainable variables

1

Hello,

I’m new to the PyTorch, and I’m trying to build a network with some additional trainable variables.

#===============================================================================
# Nnet
#===============================================================================
class Nnet(nn.Module):

    def __init__(self, layers):
        super(Nnet, self).__init__()
        self.weights = []
        self.biases = []
        num_layers = len(layers)
        
        for l in range(num_layers-1):
            W = nn.Parameter(torch.nn.init.xavier_uniform_([layers[l], layers[l+1]], gain=1.0))
            b = nn.Parameter(torch.zeros([1,layers[l+1]], dtype=np.float32))
            self.weights.append(W)
            self.biases.append(b)

                
        self.p = nn.Parameter([0.0])
        self.K = nn.Parameter([10.0])


    def forward(self, t, y):
        H = torch.cat([y,t],axis=-1)

        for l in range(len(self.weights)-1):
            W = self.weights[l]
            b = self.biases[l]
            H = torch.tanh(torch.add(torch.matmul(H, W), b))
            
        W = self.weights[-1]
        b = self.biases[-1]
        Y = torch.add(torch.matmul(H, W), b)
        
        return Y

layers = [2, 20, 20, 20, 20, 1]

nnet = Nnet(layers)

nnet = nnet.to(device)

I’m receiving he following error:

AttributeError: ‘list’ object has no attribute ‘dim’

I would appreciate it if anybody could help me.

2

The error is raised since torch.nn.init.xavier_uniform_ expects a tensor as its input, not a list of sizes.
Use:

W = nn.Parameter(torch.nn.init.xavier_uniform_(torch.empty([layers[l], layers[l+1]]), gain=1.0))

and the error should be solved.
However, since you are using plain Python lists to store the parameters, they won’t be registered properly, so use nn.ParameterList for self.weights and self.biases instead.

Once this error is solved, you will run into an argument error in b = ..., since you are passing np.float32 to a PyTorch tensor. Use:

b = nn.Parameter(torch.zeros([1,layers[l+1]], dtype=torch.float32))

and it should work.

Afterwards, self.p and self.K will fail, since nn.Parameter expects a tensor, not a list of floats.

3

@ptrblck Thank you very much for your helpful answer.
It solved my problem.