I have a little question, that I think is easy, but I did not find the answer.
I train some neural network for classification
I have a matrix A of shape (N,N), let’s say that I use for operations in the network.
I want it to be learned, except that I want only some fixed indices to be trainable, and the others to remain zero.
So, I have a support included in [0,N]x[0,N] , which is a list of indices where the coefficents of A should be learnt, and the rest is zero :
A[support] = trainable
A[not support] = zero
So how should I code this ?
I was considering creating some scalar parameters, and then assigning them into A, which would be a Variable
Is this the way to go ?
EDIT : it juste came to my mind that you can just initiate : A[support]=random, A[not support] = 0, because then the gradients with respect to the zero indices will always be zero. But that’s not a clean way to do it