Remove FC of 2 models and torch cat

vision
1

Hy guys, I have two model.
The first is a compound model with fan_resnet50 and a reset ad hoc with 4 basicBlock.
The second model is a resnet50 pretrained.

I must remove the FC layer of these models and do a torch.cat of this 2 model.

> class Identity(nn.Module):
>     def __init__(self):
>         super().__init__()
> 
>     def forward(self, x):
>         return x
> class Compound_Model_final(nn.Module):
>     def __init__(self):
>         super().__init__()
>         self.model1 = fcn_resnet50(num_classes=9)
>         self.model2= _resnet_shallow(pretrained=False)
>         self.model3 = models.resnet50(pretrained=False)
>         self.model3.load_state_dict(torch.load("Regressione_ResNet50_cartesian_angle_M95.pth"))
>     def forward(self, x):
>         dist = self.model1(x)['out']
>         #dist = torch.softmax(dist.view(dist.shape[0],-1), dim =1).view(dist.shape)
>         n_features_x = self.model2.fc.in_features
>         n_features_y =self.model3.fc.in_features
>         self.model2.fc = Identity()
>         self.model3.fc = Identity()
> 
>         z = torch.cat([self.model2.fc, self.model3.fc ],1)
>         xyuv=nn.Linear((128*(n_features_x + n_features_y)), 4)
> 
> 
> 
> 
>         return dist, xyuv

i do this wrapper class. Is it right for you?

2

You can use triple backticks ` like in github to format your code :wink:

It looks suspicious to me that your create a brand new Linear layer in the forward pass and return it, is that a typo?

3

Thanks for the reply.
Yes, the return line is:

return dist, xyuv(z)

But semantically is correct for my purpose?

4

So the problem is that because you create this linear during the forward, it will have new random weights at each forward and it’s weights are not passed to the optimizer when you do model.parameters().
Maybe that is what you want.

If you don’t want that and want the Linear to be learnt, you need to push its creation to the __init__ function and have a self.fc = nn.Linear((128*(n_features_x + n_features_y)), 4) and change your forward to do self.fc(z).

5 
class Compound_Model_final(nn.Module):

    def __init__(self):
        super().__init__()

        self.model1 = fcn_resnet50(num_classes=9)
        self.model2= _resnet_shallow(pretrained=False)

        self.model3 = models.resnet50(pretrained=False)

        self.model3.load_state_dict(torch.load("Regressione_ResNet50_cartesian_angle_M95.pth"))

        n_features_x = self.model2.fc.in_features
        n_features_y = self.model3.fc.in_features

        self.fc = nn.Linear((128 * (n_features_x + n_features_y)), 4)

    def forward(self, x):

        dist = self.model1(x)['out']
        #dist = torch.softmax(dist.view(dist.shape[0],-1), dim =1).view(dist.shape)

        
        self.model2.fc = Identity()


        self.model3.fc = Identity()

        z = torch.cat([self.model2.fc, self.model3.fc ],1)
        




        return dist, self.fc(z)

Like this?

6

You should keep in mind that the forward method in pytorch is actually executing this code every time you do a forward with a given input.
So setting the fc models to identity can be done only once in the __init__ as well.
The cat operation takes Tensors as input, not layers.
I think you meant something like, assuming x is the input for both model 2 and 3:

model2_out = self.model2(x)
model3_out = self.model3(x)

z = torch.cat([model2_out, model3_out], 1)

return dist, self.fc(z)
7

Thanks a lot :slight_smile:
I passed the identity layers in the init functions and in the forward I accepted your suggestion .

1 Like