How to train an ensemble of two CNNs and a classifier?

I have two CNNs where each CNN runs on a different dataset. The output feature vectors of these CNNs (basically without the last fc layers) are concatenated and given to a classifier. How do I train such a network?

4 Likes

How do I train such a network?

what are you exactly looking for? a pointer or a full blown example?
Generally, you can write your autograd expressions in free form and call backward on the final loss of the classifier.

I didn’t realize that my Net class could take 2 inputs and have two parallel networks inside it in free form. Thank you for the help xD

I also want to ensemble two CNNs for jointly training. May you share some code examples?
Thanks!

Here’s a basic example: (sorry for the late reply, was out of town)

class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.pool = nn.MaxPool2d(2, 2)

    #1st net
    self.conv1 = nn.Conv2d(3, 64, 3, padding = 1)
    self.conv2 = nn.Conv2d(64, 128, 3, padding = 1)
    self.conv3 = nn.Conv2d(128, 256, 3, padding = 1)
    self.conv4 = nn.Conv2d(256, 256, 3, padding = 1)
    self.conv5 = nn.Conv2d(256, 512, 3, padding = 1)
    self.conv6 = nn.Conv2d(512, 512, 3, padding = 1)
    self.conv7 = nn.Conv2d(512, 512, 3, padding = 1)
    self.conv8 = nn.Conv2d(512, 512, 3, padding = 1)

    self.fc1 = nn.Linear(512*9, 4096)
    self.fc2 = nn.Linear(4096, 4096)

    #2nd net
    self.conv1o = nn.Conv2d(1, 3, 2)
    self.conv2o = nn.Conv2d(3, 3, 2)
    self.fc1o = nn.Linear(16 * 5 * 5, 120)
    self.fc2o = nn.Linear(120, 84)

  def forward(self, x, y):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = F.relu(self.conv3(x))
    x = self.pool(F.relu(self.conv4(x)))
    x = F.relu(self.conv5(x))
    x = self.pool(F.relu(self.conv6(x)))
    x = F.relu(self.conv7(x))
    x = self.pool(F.relu(self.conv8(x)))

    x = x.view(-1, 512*9) #can also do x.view(-1, 1)

    y = F.relu(self.conv1o(y))
    y = F.relu(self.conv2o(y))
    y = x.view(-1, 1)
    y = F.relu(self.fc1o(y))
    y = F.relu(self.fc2o(y))

    x = F.relu(self.fc1(x))
    x = F.dropout(x, training = self.training)
    x = F.relu(self.fc2(x))
    x = F.dropout(x, training = self.training)

    x = torch.cat((x, y))

    return F.log_softmax(x)
5 Likes

Your advice is so instructive to me. However, I wonder about how to copy with the input training set because of the two inputs. Please reply me when you are not busy. Thanks for your attention.

I’m sorry, I didn’t understand your question. Did you mean how to give two inputs into the model? I use two datasets here, one for each sub model.

Is there a full blown example of how to ensemble two models for example- resnet152 and densenet161 for prediction on the same dataset? I would really appreciate it.

I dont have a full-blown example, but really it’s as simple as something like:

for input, targets in test_loader:
    out_resnet = resnet(input)
    out_densenet = densenet(input)
    out = out_resnet + out_densenet
    loss = loss_fn(out, target)
    total_loss += loss.item()
8 Likes

Thank you so much @smth. Appreciate your quick response. Is it possible to save these two models in one checkpoint for submission? I am participating in a challenge that requires me to load a saved checkpoint as a part of the submission.

Sure, something like this:

torch.save([densenet.state_dict(), resnet.state_dict()], 'checkpoint.pt')
densenet_state, resnet_state = torch.load('checkpoint.pt')
densenet = DenseNet(...).load_state_dict(densenet_state)
resnet = Resnet(...).load_state_dict(resnet_state)
4 Likes

Thanks again @smth. Can we do something like in this post - Combining Trained Models in PyTorch but for let’s say resnet152 and densenet161? An example would help. Appreciate your humble support.