Merging 3 models

I am trying to implemnt this model. I have the following code:

class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25))  
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)
        
    def forward(self, x):
        x = self.fc1(x)
        return x
    

    class MyModelB(nn.Module):
    def __init__(self):
        super(MyModelB, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25)) 
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)
        
    def forward(self, x):
        x = self.fc1(x)
        return x
    
     class MyModelC(nn.Module):
        def __init__(self):
        super(MyModelC, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25))  
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)

    class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA

Now i have to merge this three classes (MyModelA, MyModelB and MyModelC) into one. Do i use
tensor.cat(MyModelA, MyModelB and MyModelC) or is there something else that i should do to merge this 3 models ?

Please note the code is not complete, i am editing still. Thanks to @ptrblck for the model converging cocept code.

You would merge the model output activations inside MyEnsemble.
E.g. this code snippet removes the last linear layer of both passed models, combines their activation output, and passes them to a new linear layer, which now acts as the classification output:

class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB, nb_classes=10):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer, if necessary
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
        self.classifier = nn.Linear(2048+512, nb_classes)
        
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x

Your code contains some errors, e.g. undefined self.fc etc., but since it’s work in progress, I assume you are still fixing these. :wink:

1 Like

Thanks for your reply, @ptrblck. As you can see i have three models to concatenate to the fc layer. So at forward propagation i add another x3, is that correct ? Can torch.cat concatenate more than 2 models? Also in

self.classifier = nn.Linear(2048+512, nb_classes)

How you determined the 2048+512 ? If i add my third model what will be the number for this classifier?
Also I have defined the activaiton function inside my model, is it necessary to add the F.relu() again at the end?

torch.cat concatenates tensors. In my example the output activations.
You can pass a list of tensors (also more than two) to concatenate them in a specified dimension.

The number of input features is determined by the feature size of each model output.
E.g. if you have 3 models and each returns an activation of [batch_size, 100], you would use in_features=300 in the next layer after concatenating these tensors.