Merge layer problem

vision
1

I want to use pre-training resnet50 model to do classification problem.
But my data set not only images, but also variable term.
I get the 2048 neurons after remove last layer now.
How to merge the variable term(I have 16) to the 2048 neurons and then get 2064 neurons?
And do fully connected layer to my output layer(My prediction is 2).

Here is my pytorch code:

import torch
from torch import nn
from torchvision import models as modelset

#Pretraining model
resnet50 = modelset.resnet50()

#Remove output layer
#Known the forward will return 2048 neurons
model = nn.ModuleList(list(resnet50.children())[:-1])

#I want to merge 16 neurons and 2048 neurons
#Then the return become 2064 neurons
#And Use the 2064 neurons do fc to 2 output.

Actually, I have keras code, but don’t known how to code in pytorch,
here is keras code:

def CustomResNet():
Net = CreateResNet(include_top=True, weights=“imagenet”, input_tensor=None,
input_shape=(224,224,3), pooling=None, classes=1000)
ImageInput = Net.input
ImageOutput = Net.layers[-2].output
ImageNet = Model(inputs=ImageInput, outputs=ImageOutput)

VariableInput = Input(shape=(16,))
ModelNet = keras.layers.concatenate([ImageNet.output, VariableInput])
PredictOutput = Dense(2, activation ='softmax')(ModelNet)
TheModel = Model(inputs=[ImageInput, VariableInput], outputs=PredictOutput)
return(TheModel)
2

I think the easiest way would be to derive your custom model from torchvision.models.resnet and manipulate the forward method:

from torchvision.models import resnet

class MyResNet50(resnet.ResNet):
    def __init__(self, pretrained=False, **kwargs):
        super(MyResNet50, self).__init__(resnet.Bottleneck, [3, 4, 6, 3], **kwargs)
        if pretrained:
            self.load_state_dict(resnet.model_zoo.load_url(resnet.model_urls['resnet50']))

        self.fc = nn.Linear(2064, 2)

    def forward(self, x, y):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = torch.cat((x, y), 1)
        x = self.fc(x)
        return x
    
x = torch.randn(1, 3, 224, 224)
y = torch.randn(1, 16)
model = MyResNet50(pretrained=True)
output = model(x, y)
output.shape
3

Yes, I find your answer in the past post, thank you.