Partial feedforward network

How can I define a network like the following?
I want two or more LSTM networks (here one has 3 and other 4 states), some of these states could be related (equal letters here). I want to connect them to a partial feedforward layer and finally output some outputs (here 5 equal to distinct letters).

A  --------------A
B  ----------|--B
C  -------|-----C  
        |     |
B-----|     |
C ---------|
D  -----------D
E  ---------- E

Could you please guide me, I am novice with Pytorch