Append feature maps to intermediate layers of network

vision
1

I have the following network architecture shown below. It takes in input of size 16x8x2048, and gives an output of size 256x128x3. I have a feature map which I wish to append along the channel dimension after every upsampling layer in the below architecture. I would be scaling the feature maps to the appropriate resolution before appending. How can I append these feature maps in the intermediate layers of the network?

model = []
model += [Conv2dBlock(2048, 256, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(256, 128, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(128, 64, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]                        
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(64, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(32, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]                        
model += [Conv2dBlock(32, 3, 3, 1, 1, norm='none', activation=activ, pad_type=pad_type)]                        
model = nn.Sequential(*model)