Hi, sorry for the inconvenience, I’m new to pytorch.
I would like to build a custom network that integrates two types of data (tabular and image). In this network, the output of a fully connected layer (tabular data input) multiplies the output of a convolutional network layers. For this, the number of neurons in the output is equal to the number of channels in the conv network (channel wise multiplication).
my init looks like this (note: i have a Conv function outside the class.)
self.conv1 = conv_block(3, 16)
self.conv2 = conv_block(16, 32)
self.conv3 = conv_block(32, 64)
self.ln1 = nn.Linear(64 * 26 * 26, 16)
self.relu = nn.ReLU()
self.batchnorm = nn.BatchNorm1d(16) #Na predição eliminar o Batch
self.dropout = nn.Dropout2d(0.5)
self.ln2 = nn.Linear(16, 3)
self.ln4 = nn.Linear(5, 16)
self.ln5 = nn.Linear(16, 16)
self.ln6 = nn.Linear(16, 32)
self.ln7 = nn.Linear(32, 64)
self.relu = nn.ReLU()
and forward pass like this:
def forward(self, img, tab):
tab = self.ln4(tab)
tab = self.relu(tab)
tab = self.ln5(tab)
tab = self.relu(tab)
#Entra a Primeira Mul
img = self.conv1(img)
img.mul(tab[1:])
tab = self.ln6(tab)
tab = self.relu(tab)
#Entra a Segunda Mul
img = self.conv2(img)
img = torch.mul(img, tab)
tab = self.ln7(tab)
tab = self.relu(tab)
#Entra a Terceira Mul
img = self.conv3(img)
img = torch.mul(img, tab)
img = img.reshape(img.shape[0], -1)
img = self.ln1(img)
img = self.relu(img)
img = self.batchnorm(img)
img = self.dropout(img)
img = self.ln2(img)
return img
but everything i try is generating incompatibility in multiplication
RuntimeError: The size of tensor a (111) must match the size of tensor b (31) at non-singleton dimension 2
just for curiosity, the initial multiplications are Image Conv Layer [32, 16, 111, 111] and
Tabular Output first linear ([32, 16])
Can someone help me. It works on Keras using multiply method.