Is there a way to have one weight matrix that is partly trainable and partly not?
Im trying to implement a RNN module that incorporates multiple time scales (as proposed in A Clockwerk RNN) but struggle on weight matrices that are partitioned in blocks of trainable weights and untrainable zeros (W_H in the image below).
I tried to just initialize the Parameters as individual blocks and concatenate them, but using torch.cat on Parameters doesn’t work.
Is there a nice way of handling this or do I have to do all calculations separately?