How to create an optimizer given a loss function

autograd
1

I’m quite confused about how the optimizer created. From the Pytorch tutorial, when people implement SGD by defining a custom module first, which contains a constructor and a forward function, then using SGD optimizer to calculate the model parameters (please see the following code). My question is what if I am only given a loss function(please see the following pic) and I don’t have a “forward” function, how to create an optimizer? If my question is not clearly stated, please let me know and I’ll provide further information, thanks for any help in advance.

import torch

class model(torch.nn.Module):

def __init__(self, D_in, H, D_out):
    super(model, self).__init__()
    self.linear1 = torch.nn.Linear(D_in, H)
    self.linear2 = torch.nn.Linear(H, D_out)

def forward(self, x):
    h_relu = self.linear1(x).clamp(min=0)
    y_pred = self.linear2(h_relu)
    return y_pred

model = model(D_in, H, D_out)

optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)

Screen Shot 2020-02-22 at 2.47.51 PM
Screen Shot 2020-02-22 at 2.47.51 PM872×164 6.57 KB

2

The optimizer just stored the references to the passed parameters and uses their .grad attribute to perform the update step.
It does not depend on the forward pass.
If you are able to generate the gradients in some way, the optimizer will just use them via their .grad.

3

So basically what you mean is that I can directly call “torch.optim.SGD(parameter.grad, lr = 1e-4)” something like that?

4

No, you still need to pass the parameters as:

torch.optim.SGD(model.parameters(), lr=1e-4)
# or as a list
torch.optim.SGD([param1, param2], lr=1e-4)

The optimizer will then use the param.grad attributes to update them.

5

Thank you for your kindly help! It helps me to understand Pytorch a lot!