So I’m trying to implement a deep neural network composed of a few linear layers. Every layer is composed of a 2x2 weight matrix and no biases. The peculiarity of my network is that I would only like to train one of the elements of each weight and leave the others untouched.
To make things more concrete, every weight looks like:
W = torch.tensor([[number1, number2],[number3, part I want to train]])
Things I’ve thought is proceed as usual (i.e. calculate the loss function and then use some optimizer) and then somehow zero the gradients for the parts of the weights I don’t want to touch but I haven’t managed to succeed. Any ideas? I can share parts of the code if that helps