Channel-wise multiplication Conv Layer and Linear Layer

Hi, sorry for the inconvenience, I’m new to pytorch.

I would like to build a custom network that integrates two types of data (tabular and image). In this network, the output of a fully connected layer (tabular data input) multiplies the output of a convolutional network layers. For this, the number of neurons in the output is equal to the number of channels in the conv network (channel wise multiplication).

my init looks like this (note: i have a Conv function outside the class.)

        self.conv1 = conv_block(3, 16)

        self.conv2 = conv_block(16, 32)

        self.conv3 = conv_block(32, 64)

        self.ln1 = nn.Linear(64 * 26 * 26, 16)

        self.relu = nn.ReLU()

        self.batchnorm = nn.BatchNorm1d(16)    #Na predição eliminar o Batch

        self.dropout = nn.Dropout2d(0.5)

        self.ln2 = nn.Linear(16, 3)      

        self.ln4 = nn.Linear(5, 16)

        self.ln5 = nn.Linear(16, 16)     

        self.ln6 = nn.Linear(16, 32)    

        self.ln7 = nn.Linear(32, 64)   

        self.relu = nn.ReLU()

and forward pass like this:

def forward(self, img, tab):

        tab = self.ln4(tab)

        tab = self.relu(tab)

        tab = self.ln5(tab)

        tab = self.relu(tab)

        #Entra a Primeira Mul

        img = self.conv1(img)

        tab = self.ln6(tab)

        tab = self.relu(tab)

        #Entra a Segunda Mul

        img = self.conv2(img)

        img = torch.mul(img, tab)

        tab = self.ln7(tab)

        tab = self.relu(tab)

        #Entra a Terceira Mul 

        img = self.conv3(img)

        img = torch.mul(img, tab)

        img = img.reshape(img.shape[0], -1)

        img = self.ln1(img)

        img = self.relu(img)

        img = self.batchnorm(img)

        img = self.dropout(img)

        img = self.ln2(img)
        return img

but everything i try is generating incompatibility in multiplication

RuntimeError: The size of tensor a (111) must match the size of tensor b (31) at non-singleton dimension 2

just for curiosity, the initial multiplications are Image Conv Layer [32, 16, 111, 111] and
Tabular Output first linear ([32, 16])

Can someone help me. It works on Keras using multiply method.

You would need to unsqueeze the missing dimension so that PyTorch can broadcast the smaller tensor:

conv_output = torch.randn(32, 16, 111, 111)
a = torch.randn(32, 16)

out = conv_output * a
# > RuntimeError: The size of tensor a (111) must match the size of tensor b (16) at non-singleton dimension 3

out = conv_output * a[:, :, None, None] # works
1 Like

It worked. The solution worked well. Thank you so much!