Can not update parameters and .grad is none

autograd
1

Hello,

I am trying to train my model and my code is listed below:

import torch.nn.functional as F

data = Variable(data,requires_grad=True)
optimizer.zero_grad()
output = model(data)
predict = F.softmax(output,0)
final_predict = 1.2 * predict[0] + 1.3 * predict[1] + 2.5 * predict[2]
loss = F.mse_loss(final_predict, ground_truth)
loss.backward()

print(predict.grad)

However, the parameters of my model remain the same and the gradient is none.
So, how to solve this issue? Thanks.

2

Your predict is not a parameter so it by default doesn’t retain gradients.

3

How should I modify this code?

4
  1. As said above, predict doesn’t have .grad because it is not a parameter, but only an intermediate computation result. The actual parameters of your model will have .grad calculated.
  2. To update the model use these .grad, you would need an optimizer. There are a bunch of options and examples here: http://pytorch.org/docs/master/optim.html
  3. You don’t need requires_grad=True on your data unless you need the gradients with respect to the data.
5

Thanks.

Now, I slightly modify my code:

optimizer = optim.SGD(model.parameters(), lr = lr, momentum = momentum)
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.classifier = nn.Sequential(
            ......
            ......
            nn.Softmax(0),
        )

    def forward(self, x):
        output = self.classifier(x)
        final_predict = 1.2 * output[0] + 1.3 * output[1] + 2.5 * output[2]
        return final_predict

predict = model(data)
loss = F.mse_loss(predict, ground_truth)
loss.backward()

Obviously, the optimizer still can not see final_predict. So, how can I add final_predict to the existing optimizer or convert it to a nn.parameter()? Because I need to use final_predict to calculate the loss so as to update my model.

6

Did you read the examples in the link I gave above?