Learnable parameters

vision
1

I want to train learnable parameter w to re-weight loss.
But w is never changed.
What`s wrong?

w = nn.Parameter(torch.tensor(0.1), requires_grad=True).cuda()
optimizer = optim.SGD(list(network.parameters()), lr=0.1, momentum=0.9, weight_decay=5e-4)
...
loss = loss * w
optimizer.zero_grad()
loss.backward()
optimizer.step()
2

w doesn’t seem to be passed to the optimizer, so that even if this parameter gets valid gradients, it won’t be updated.
Also, call cuda() on the tensor, not the nn.Parameter, as this operation will create a non-leaf variable.

1 Like
3

@ptrblck how to update that to pass it to the optimizer?

4

You could pass w as an element of a list to the optimizer:

optimizer = torch.optim.SGD([w], lr=0.1)

Of course concatenating w with all other model parameters into a list would also work.

1 Like
5

Hi! It’s not clear to me, how w and the list of model parameters can be concatenated?

Best,
Dimitri

6

Hi @DK13,

You can try something like

params = [p for p in model.parameters()]
params.append(w)
optim = torch.optim.Adam(params)

This is without moving the tensors to cuda but just move them before you pass them to the optimizer and all should work! :slight_smile:

7

Thanks a lot! I tried also list(model.parameters())+[w], I think it does the same thing.