Manually feeding trainable parameters to the optimizer

1

Suppose I don’t use torch.nn.Module and I have some trainable weight matrices w1 and w2. Then how do I feed these parameters into the optimiser? Would I do something like? :

torch.optim.SGD({w1,w2}, lr=1e-4)

2

The first argument you feed when initializing any optimizer should be an iterable of Variables or dictionaries containing Variables. In your case a simple list containing w1 and w2 should be fine as long as those are Variables that require gradients.

import torch
import torch.optim as optim
from torch.autograd import Variable

w = Variable(torch.randn(3, 5), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)
x = Variable(torch.randn(5))

optimizer = optim.SGD([w,b], lr=0.01)

optimizer.zero_grad()
y = torch.mv(w, x) + b
y.backward(torch.randn(3))
optimizer.step()
5 Likes