So basically I’d like to define a layer that takes in a fixed kernel.
For example:
class FixedConv2d(nn.Module):
def __init__(self):
super(FixedConv2d, self).__init__()
f = np.random.rand(5, 5).astype(np.float32)
f = f.reshape(1, 1, f.shape[0], f.shape[1])
f = np.repeat(f, 100, axis=1)
f = np.repeat(f, 100, axis=0)
self.f = nn.Parameter(data=torch.FloatTensor(f), requires_grad=False)
def forward(self, x):
return F.conv2d(x, self.f)
I added this module to my network using nn.Sequential(). I don’t want it to be optimized by the optimizer. When I pass the paramters of my network (mode.paramters()) to the optimizer, it complains because self.f doesn’t require grad.
I’m wondering what’s the correct way of doing this?
I would probably try to use a Variable instead of a Parameter, that should make things ignore it completely, which seems to be what you want.
Or you could do layer.f.requires_grad = False after initializing the optimizer. Then the gradient will not be computed, the optimizer to see 0 all the time. (But the optimizer would still do “computation”.)
Thanks for the reply! The latter option would probably work.
I tried using a Variable, but the tricky thing is that a Variable in a module won’t respond to the cuda() call (Variable doesn’t show up in the parameter list, so calling model.cuda() does not transfer the Variable to GPU). Note that overriding the cuda() method won’t work either.
I found a better way to solve this problem.
we can still use nn.Parameter, but exclude it from the parameter list by overriding the named_parameters() method:
Overriding parameters() won’t work because only named_parameters() is being called recursively (if you want to use the customized module as a layer in a network).
I’d probably stay away from messing with the internal functions.
A cleaner option may be to filter the parameters passed to the optimizer, prefix fixed params with fix_ and instantiate the optimizer like SGD([p for n,p in m.named_parameters() if not n.startswith('fix_')], lr=my_lr).