LSTM late fusion time series prediction

Greetings to everyone!

I have a dataset with 4 modalities and I want to feed each modality to a LSTM or RNN and then combining the results of all the 4 networks in order to obtain a global result.

I wrote a couple of examples with RNN and softmax at the end, but I don’t know how to deal with 4 networks at once.
Does anyone knows a simple example of combining 2 LSTM’s or RNN’s by means of late fusion in PyTorch?
Should I feed all the 4 networks to the same loss function?
Thanks a lot!


Did you find how this late fusion is feasible? Since I am facing the same issue.