Update Weights With Learnable Parameters

Hi all,

I am trying to implement a convolutional neural network where one of the layers’ weights are constrained by a couple of parameters, which are optimized during backpropagation. Another way to put it: the weights themselves are not directly updated, but the parameters that determine the weight matrices should be. Here is a simplified version of the code:

import torch.nn.functional as F
import torch
from torch import nn

class GaborConv2d(nn.Module):
    def __init__(

        self.is_calculated = False

        self.conv_layer = nn.Conv2d(

        self.kernel_size = self.conv_layer.kernel_size
        self.test = nn.Parameter((torch.Tensor([2.0])), requires_grad=True)
        self.register_parameter("test", self.test)

    def forward(self, input_tensor):
        if self.training:
            self.is_calculated = False
        if not self.training:
            if not self.is_calculated:
                self.is_calculated = True

        return self.conv_layer(input_tensor)

    def calculate_weights(self):
        self.conv_layer.weight.data += self.test

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = GaborConv2d(1, 1, 3)
        self.fc = nn.Linear(9, 3)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        x = F.relu(x)
        return x

if __name__ == '__main__':
    model = Net()
    model.optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9) # only the weights and biases of the fully connected layer are learned
    model.loss = nn.CrossEntropyLoss()
    for i in range(100):
        random_data = torch.rand((1, 1, 5, 5))
        output = model(random_data)
        l = model.loss(output, torch.tensor([0]))

The “test” variable is the parameter that controls the weights. In the optimizer, I do see test listed in the params group. However, its grad is None, and it is not updated during backpropagation. Any input would be appreciated! Thanks!


I don’t think you need to re-register test after declaring it as an nn.Parameter object. So, delete the line,

self.register_parameter("test", self.test)

Secondly, within your calculate_weights method there are 2 problems. Firstly, don’t use the .data attribute as it’s deprecated and can lead to unexpected behaviour. And, you’re updating conv_weight.weight via an inplace operator (+=). So try,

self.conv_layer.weight = self.conv_layer.weight + self.test 

Thank you for your input!

self.conv_layer.weight = self.conv_layer.weight + self.test gave me this error:
TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected), so I tried
self.conv_layer.weight = nn.Parameter(self.conv_layer.weight + self.test) and
self.conv_layer.weight = nn.Parameter(self.conv_layer.weight + self.test, requires_grad=False). In both cases, the grad’s and grad_fn’s for “test” are still None. I also added a line before backward() to retain test’s gradient: model.conv1.test.retain_grad(). The issues persist.

Hi Lillian!

Could pytorch’s parametrizations facility be applied to your use case?


K. Frank