My learnable vector cannot be updated

I want a learnable vector in ResNet to get
In init function of ResNet, I insert some code.

        weight = torch.rand((5, 1), device='cuda:0')
        self.weight = weight.requires_grad_(True)

And in forward function, I use matrix multiplication between out and self.weight
out is a result of original ResNet. (self._forward_impl)

    def forward(self, x):
        out = self._forward_impl(x)                 #(batch_size * 5, 1)

        out = out.view(-1, 5)                       #(batch_size, 5)
        out = torch.mm(out, self.weight)            #(batch_size, 5)x(5, 1)=(batch_size, 1)

        return out

When I get a loss value using Focal Loss and run loss.backward(), it doesn’t change the self.weight value and sometimes self.weight.grad is None

print(self.weight)
print(self.weight.grad)
tensor([[0.2026],
        [0.6027],
        [0.2351],
        [0.0260],
        [0.2805]], device='cuda:0', requires_grad=True)
tensor([[-0.1866],
        [-0.2713],
        [-0.2793],
        [-0.1892],
        [-0.1899]], device='cuda:0')
tensor([[0.2026],
        [0.6027],
        [0.2351],
        [0.0260],
        [0.2805]], device='cuda:0', requires_grad=True)
tensor([[-0.2546],
        [-0.3381],
        [-0.3341],
        [-0.2421],
        [-0.2273]], device='cuda:0')

I don’t know why self.weight cannot be updated…

Wrap your weight with nn.Parameter. After that, the tensor is registered as one of model parameters. Otherwise, it is not visible to model.parameters(), which are optimizer tries to update.

2 Likes

Thank you! I change my code as you said, now it works well!