Which is the best practice in coding PANET style networks?

I am currently implementing from scratch YOLOV5 and I was wondering if there are any suggestion on how to write down “elegantly” such networks.

In my case the only strategy that comes to my mind is to for code separately each top-down/buttom-up routes (the arrows in the picture) to store the desired routes_connections tensors to be used later on.
More precisely am creating separated nn.Sequential() or nn.ModuleList() for each one of those blocks.

However I was wondering if any of you know if it is possible to link the parts 2) and 3) together inside a unique nn.Sequential or nn.ModuleList()

Hope I was clear,