Fusion/ensembel of two MLP models

1

Hi

I’m trying to combine the features of two MLP models.

I have two tabular datasets of the same class that I want to fuse to improve accuracy. The straightforward approach is currently considering concatenation.

Here is a summary of my two models; both models are identical except for the input features.

MulticlassClassificationA(
(layer_1): Linear(in_features=4, out_features=1500, bias=True)
(layer_2): Linear(in_features=1500, out_features=1000, bias=True)
(layer_3): Linear(in_features=1000, out_features=500, bias=True)
(layer_out): Linear(in_features=500, out_features=3, bias=True)
(relu): ReLU()
(dropout): Dropout(p=0.3, inplace=False)
(batchnorm1): BatchNorm1d(1500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(batchnorm2): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(batchnorm3): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
MulticlassClassificationB(
(layer_1): Linear(in_features=78, out_features=1500, bias=True)
(layer_2): Linear(in_features=1500, out_features=1000, bias=True)
(layer_3): Linear(in_features=1000, out_features=500, bias=True)
(layer_out): Linear(in_features=500, out_features=3, bias=True)
(relu): ReLU()
(dropout): Dropout(p=0.3, inplace=False)
(batchnorm1): BatchNorm1d(1500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(batchnorm2): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(batchnorm3): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)

I tried the code posted by @ptrblck, the ensembele model, but it did not work.

class MyEnsemble(nn.Module):
def init(self, model_EHR, model_G, nb_classes=3):
super(MyEnsemble, self).init()
self.model_EHR = model_EHR
self.model_G = model_G
# Remove last linear layer
self.model_EHR.layer_out = nn.Identity()
self.model_G.layer_out = nn.Identity()

    # Create new classifier
    self.classifier = nn.Linear(500+500, nb_classes)
    
def forward(self,x):
    x1 = self.model_EHR(x.clone())  # clone to make sure x is not changed by inplace methods
    x1 = x1.view(x1.size(0), -1)
    x2 = self.model_G(x2)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    
    x = self.classifier(F.relu(x))
    return x

for param in model_EHR.parameters():
param.requires_grad_(False)

for param in model_G.parameters():
param.requires_grad_(False)

Create ensemble model

model = MyEnsemble(model_EHR, model_G)

another question: what input will I give the model when I do the ensemble model?

Is it a batch from dataset A’s loader and a batch from dataset B’s loader?

I’m not sure how I’ll approach the prediction after concatenation?

Any help is greatly appreciated!

2

In my example you can see that the same input is passed to both models so that the features created by these models are then concatenated before being passed to a final classifier.
In your code snippet you are passing an unknown x2 tensor to self.model_G so I guess this might be a typo?

3

Thanks a lot for the quick response, you are right! it was a typo. Yet It didn’t work.

this is the error:

TypeError Traceback (most recent call last)
in
7 X_batch = X_batch.to(device)
8 X_batch_G = X_batch_G.to(device)
----> 9 y_test_pred_F = model(X_batch,X_batch_G)
10 #probs = torch.nn.functional.softmax(y_test_pred, dim=1)
11 #y_prob=y_prob.np.append(probs.detach().cpu().numpy())

~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
→ 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() takes 2 positional arguments but 3 were given

Let me tell you about my dataset a bit . I have a CVS file for the EHR model which has 4 features (col) and it is my x_train (Features:age, gender, censor,survive) the table is of 742 records (rows)

The other CSV file for training the model_G has 78 features which is my x_train_G, this table again is of 742 records

the label for both models are the same

As I mentioned previously the two models are identical but have different input shapes

this is the code snippet that brings the error

y_pred_list_Fusion = []

with torch.no_grad():
model.eval() # Ensembel model
for X_batch, _ in test_loader: # test loader of EHR data
for X_batch_G, _ in test_loader_G: # test loader of Genes data
X_batch = X_batch.to(device)
X_batch_G = X_batch_G.to(device)
y_test_pred_F = model(X_batch,X_batch_G)
_, y_pred_tags = torch.max(y_test_pred_F, dim = 1)
y_pred_list_Fusion.append(y_pred_tags.cpu().numpy())

y_pred_list_Fusion = [a.squeeze().tolist() for a in y_pred_list_Fusion]

I would be happy to have your kind feedback!

4

You re passing two inputs to the model:

y_test_pred_F = model(X_batch,X_batch_G)

while a single one is expected:

def forward(self,x):

so either pass a single input or change the forward method.

PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.

5

Again thanks for your prompt response!

Here is my confusion. When I send the data for the forward training (using ensemble model) at this point

y_test_pred_F = model(X_batch,X_batch_G)

am I doing something wrong?

I want to send the data from both (CSV) files that’s why I sent two patches, please correct if I am wrong.

When did you say pass one input, how to do that in my case? I do need 2 inputs from two datasets?

How to change the forward method, please?

I am sorry I am very new to PyTorch and still learning :slight_smile:

6

Since you want to pass both inputs to the model, I guess your forward method should be:

def forward(self, x1, x2):
    x1 = self.model_EHR(x1)
    x1 = x1.view(x1.size(0), -1)

    x2 = self.model_G(x2)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    
    x = self.classifier(F.relu(x))
    return x

as this will now pass the two different inputs to the two internal models.

7

Many thanks for your prompt response!

It worked :slight_smile: