"Learnable" parameter does not want to learn

I am trying to use “mult” as learnable weights in this model, yet despite being in model.parameters() and requires_grad=True the weights aren’t being updated (and indeed mult.grad = None). What could have gone wrong?

    class NeuralNetB(nn.Module):
        def __init__(self):
            super(NeuralNetB, self).__init__()
            self.mult = nn.Parameter(torch.rand(3), requires_grad=True).to(device) #todo: why not learnable?
            self.fc1 = nn.Linear(6, 64)
            self.relu = nn.LeakyReLU()
            self.fc2 = nn.Linear(64, 16)
            self.fc3 = nn.Linear(16, 3)
            self.drop = nn.Dropout(0.4)
            self.fc4 = nn.Linear(30, 3)

        def forward(self, x, tc):
            print(self.mult)
            x2 = x/tc.view(tc.shape[0],1,1) #get partial averages
            x2 = x2*(self.mult*10).view(1,3,1) #multiply by weights
            out = torch.sum(x2, dim=1) #sum them up
            #print(self.mult)
            out = self.fc1(out)
            #out = self.relu(out)
            out = self.fc2(out)
            out = self.relu(out)
            out = self.fc3(out)
            #out = self.drop(out)
            #out = self.fc4(out)
            return out

Training:

            y_pred = model(train_xB[v], train_count[v])
            model.zero_grad()
            loss = criterion(y_pred, train_yB[v])
            loss.backward()
            optimizer.step()

Hi,

The nn.Parameter() always require gradients, no need to specify it.
Also when you do .to() you get a new Tensor (which is not a leaf Tensor !) Didn’t you get a warning when you did the optimizer step?

You should change the definition to self.mult = nn.Parameter(torch.rand(3, device=device))

1 Like

Ah that worked, thanks so much! There was no warning.