I am trying to implement the following model in torch. I have found source code for Tensorflow but not Torch. If there is any source that I can read to help me build the model in Torch that would be great.
View1 and View2 are the two inputs. Both inputs are fully connected to one of the middle layers ( the right one), and to separate neurons in the other latent space ( the left one, you can see that view 1 is connected to the right neuron in the left box, and view 2 is connected to the left neuron in the left box)
Maybe @ptrblck may help you in a better way but I think what you can do is building up the whole model as a single nn.Linear layer and to set gradients equal to 0 for the corresponding neurons in the backward pass (obviously initializing that weight as zero too)
It’s a little bit mess because weights are passed as a package, so you cannot disable requires grad in a subgraph.
Another option could be to initialize corresponding weights to 0 and detaching specific neurons during the forward pass.