Concatenating tensor in middle of sequential

Hello,
I’ve read nn.Module with multiple inputs, and this is more of a follow-up question here. I’m curious if there is a way to concatenate a tensor in the middle of sequential. Or in the middle of a model, for example:

def __init__(self, z_dim=10, nc=1,stim=np.random.randint(13,size=8)):
    super(BetaVAE_H_wStim, self).__init__()
    torch.backends.cudnn.benchmark = True
    self.z_dim = z_dim
    self.nc = nc
    self.stim = stim
    self.encoder = nn.Sequential(
        nn.Conv3d(nc, 32, 3, (1,2,2), 1),          # B, 32, 4, 48, 48
        nn.ReLU(True),
        nn.Conv3d(32, 32, 3, (1,2,2), 1),          # B, 32, 2, 24, 24
        nn.ReLU(True),
        nn.Conv3d(32, 64, 3, 2, 1),            # B, 64, 1, 12, 12
        nn.ReLU(True),
        nn.Conv3d(64, 64, 3, 2, 1),            # B,  64,  6,  6
        nn.ReLU(True),
        nn.Conv3d(64, 256, 3, 2, 1),          # B,  64,  6,  6
        nn.ReLU(True),
        View((-1,256)),                               # B, 256
        # i want to concat here
        nn.Linear(256,z_dim*2),                # B, z_dim*2
    )

Where my comment is is where I’m hoping to concat a numpy array of size 8 (1 axis) to my tensor in the model. It’s different from the problem posted above. I actually want those values to be fully connected to the output of this model.

Is there an easy way to do this?

Sure it’s possible, though only during the forward pass; meaning that you should declare your encoder and the final Linear separately. Finally during the forward pass, take the ouput of the encoder, reshape with .view, concat with the other tensor with torch.cat (https://pytorch.org/docs/stable/torch.html#torch.cat) and pass it to the Linear module.

Ok thank you very much. Along the same lines, I have a decoder that looks like this:

    self.decoder = nn.Sequential(
        nn.Linear(z_dim, 256),               # B, 256
        # I want to separate here
        View((-1, 256, 1, 1)),               # B, 256,  1,  1
        nn.ReLU(True),
        nn.ConvTranspose2d(256, 64, 4),      # B,  64,  4,  4
        nn.ReLU(True),
        nn.ConvTranspose2d(64, 64, 4, 2, 1), # B,  64,  8,  8
        nn.ReLU(True),
        nn.ConvTranspose2d(64, 32, 4, 2, 1), # B,  32, 16, 16
        nn.ReLU(True),
        nn.ConvTranspose2d(32, 32, 4, 2, 1), # B,  32, 32, 32
        nn.ReLU(True),
        nn.ConvTranspose2d(32, nc, 4, 2, 1),  # B, nc, 64, 64
    )

And I want to somehow remove that tensor I just concatenated after my 1st linear layer in the decoder. Is this likewise as simple?

@trypag any idea on how to remove that same tensor in the decoder?

Yes it’s possible, though I am confused about where the concatenation happens in the encoder as the parameters of Linear seem to be incorrect : should be nn.Linear(256+8,z_dim*2) if you concatenate with previous View. Try to make a first working model with initialization and forward, and I will help.