Parallel two encoder in Pytorch

[ptrblck]

You could use the two models in a new custom nn.Module, execute both forward passes, and concatenate the output.
This post was written for a slightly different use case, but might be a good starter.