Multihead anomaly detector using PyTorch

I want to implement the model that consists of 5 different LSTMs as inputs where 5 corresponds to number of incoming signals of different shapes and these are followed by a joint vector, 2 FC-layers and reconstruction in order to perform some anomaly detection.

I assume this could be achieved by creating 5 different models sharing their weights. At each timestamp just one signal arrives and only the corresponding weights are to be updated.
How should I approach this strategy using torch properly? Particularly I don`t understand how to associate joint vector “parts” with corresponding lstm cells?
Thank you for your attention.