Combine two model on pytorch?

Hello everyone,
How to combine two models parameter of two different datasets to generate one model like :

class NetworkA(nn.Module):
    def __init__(self, Input, Output):
        super(NetworkA, self).__init__()
    def forward(self, x):
        return x

class NetworkB(nn.Module):
    def __init__(self, Input, Output):
        super(NetworkB, self).__init__()
    def forward(self, x):
        return x

In the Training Step combine the two parameters , then, for Testing i use only the combined resulted model.
Any suggestion, ideas would be much appreciated.

3 Likes

How would you like to combine the parameters of the two models?
Could you give an example of the use case you are thinking about?

1 Like

I have datasets that has many channels(hyperspectral images) where each pixel has a cube of value “Pi(1,1,200)”, i would like to create a model for spatial features and other for spectral features for the same datasets but reshaped in different way, and then combine these models.

Thanks for the information.
In this case, you could use two submodules (each working on the specific data samples) and concatenate these features later in the model.
The concatenation might be applied e.g. on the activation volumes or the linear activation outputs.

1 Like

can you kindly provide some example oh how to concatenate them , just to get it clearly, still could not work it out.
Thanks Ptrblck for helping :blush:

Sure! Here is a small example concatenating the outputs of two linear layers:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.features1 = nn.Sequential(
            nn.Conv2d(3, 6, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(6, 12, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        
        self.features2 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(6, 12, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        
        self.fc1 = nn.Linear(12*6*6, 64)
        self.fc2 = nn.Linear(12*6*6, 64)
        
        self.fc_out = nn.Linear(128, 10)
        
    def forward(self, x1, x2):
        x1 = self.features1(x1)
        x1 = x1.view(x1.size(0), -1)
        x1 = F.relu(self.fc1(x1))
        
        x2 = self.features2(x2)
        x2 = x2.view(x2.size(0), -1)
        x2 = F.relu(self.fc2(x2))

        # Concatenate in dim1 (feature dimension)
        x = torch.cat((x1, x2), 1)
        x = self.fc_out(x)
        return x

model = MyModel()
x1 = torch.randn(2, 3, 24, 24)
x2 = torch.randn(2, 1, 24, 24)
output = model(x1, x2)

Note that you could also concatenate/sum/average the outputs of the features modules.
I’m not sure what would work best in your use case, so you might want to play around a bit. :wink:

5 Likes

Thank you so much Ptrblck , much helpful :smile:

guys I have similar issue if you could help me please.
I have two different models. I trained the first model (AE). Then, I want to feed the output of the AE into the second model. while doing that, I freeze the parameters of AE. I concatenate them similar to the above example and has no issue. my question how to make the second model (traditional cnn) look at/focus more on the features (weights) coming from AE? what is efficient way to do it? because I want the first model to improve the accuracy of my second model.

Hi, ptrblck

I try to use the structure you mentioned to build a model, but the model parameters did not change, and grad_value is None. Can you help me? Thanks!

Please see How to assemble two models into one big model - FAQ - PyTorch Forums