Hope you are having a productive day.
I am trying to build a model which has
- sparse connections between layers
- some weights that should not be updated
self.L_vc = nn.Linear(E, E, bias=False) torch.nn.init.ones_(self.L_vc.weight) for p in self.L_vc.parameters(): p.requires_grad = False prune.custom_from_mask(self.L_vc, name='weight', mask=vc_mask)
Here, I specify the connectivity between the neurons using the tensor
vc_mask and preventing the weights from being updated by setting
p.requires_grad = False. I also set
bias = False since I just want
L_vc to specify the connectivity just like an adjacency matrix. I am not sure if this is the correct way to do that.
Am I interrupting the gradients from propagating backwards by setting a constant layer like this? What should I expect if I have a layer like this in the middle of the model? Any help would be highly appreciated.