Make nn.Parameter recognizable by optimizer

1

Here is my model.

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = ...
        self.conv3 = ...
        self.custom_param = torch.nn.Parameter(torch.zeros(100, 50, requires_grad=True)).cuda()

model = Model()

When i send the mode.parameters() to my optimizer I am not able to view my custom_param parameter in optimizer.param_groups[0]['params'] this makes the custom_param weights not trainable. How do I make this parameter recognizable by optimizer ?

2

Could you try to call .cuda() on the tensor instead of the nn.Parameter?
Most likely custom_param isn’t a leaf variable anymore as it was created by .cuda().
Have a look at this post for more information.

1 Like
3

I have a similar problem and I have specified .to(device) instead of .cuda(). But I am getting error that: AttributeError: 'Tensor' object has no attribute 'parameters'

4

I guess you are calling .parameters() on a tensor instead of an nn.Module.
Make sure to call this method on the model itself and not (internal) tensors.

5

Yes I am calling the parameters of that particular tensor for specifying the learning rate something like this:
optimizer = optim.Adam([{'params':model.custom_param.parameters(), 'lr':1e-3}, ... and so on])

How can I do that?

6

A tensor doesn’t have the .parameters() method, so either call it on an nn.Module object or pass the nn.Parameter to the optimizer instead.

2 Likes
7

So inside the model class if I have defined the layer something like this: self.actor_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape))), then I can directly call optimizer = optim.Adam([{'params':model.actor_logstd, 'lr':1e-3}]) without adding the .parameters() method?

8

You are defining a parameter, not a layer.

Yes, you can pass parameters like this to the optimizer.