Insert New Layer in the Middle of a Pre-trained Model

#1

Help!!! Does anyone knows how to insert a new layer in the middel of a pre-trained model? e.g insert a new conv in the middle of Resnet’s bottelneck.

2 Likes
Dynamically insert new layer at middle of a pre-trained model
#2

There’s no easy way to insert a new layer in the middle of an existing model as far as I’m aware. A definite solution is to build the structure that you want in a new class and then copy the corresponding weights over from the pretrained model.

1 Like
#3

Figured!!!
After loading model，we can directly specify model.conv_x = nn.Sequential([new_layer, model.conv_x]), by this way, we can still use thepretrained model.conv_x

9 Likes
#4

@wangchust
Can you help a newbie like me. I import pretrained resnest34 as:

resnet = models.resnet34(pretrained=True)

Now I want to insert a conv2d 1x1 kernel layer before the fc to increase channel size from 512 to 6000 and then add a fc 6000 x 6000

I am so new to pytorch that I need some hand-holding. Can you write the lines of code needed? I am still at monkey-see monkey-learn stage. Thanks in anticipation

2 Likes
#5

I think the easiest approach would be to derive from ResNet and add your layers.
This should do what you need:


class MyResnet2(models.ResNet):
    def __init__(self, block, layers, num_classes=1000):
        super(MyResnet2, self).__init__(block, layers, num_classes)
        self.conv_feat = nn.Conv2d(in_channels=512, 
                                   out_channels=6000, 
                                   kernel_size=1)
        self.fc = nn.Linear(in_features=6000,
                            out_features=6000)
        
    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = self.conv_feat(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)

        return x

from torchvision.models.resnet import BasicBlock
model = MyResnet2(BasicBlock, [3, 4, 6, 3], 1000)
x = Variable(torch.randn(1, 3, 224, 224))
output = model(x)

Note that the shape of x in x= self.avgpool(x) is already [batch, 512, 1, 1] for x.shape = [batch, 3 ,224, 224].
You could therefore flatten x and just use two Linear layers, since it would be the same as a Conv2d with kernel_size=1.

10 Likes
#6

@ptrblck
Thank you sir!,
Now if I re-read some of the tutorials, they will register in my head.

1 Like
#7

Sorry for being so late to reply. My way should be replace ‘Resnet.fc = nn.Linear(512, num_classes)’ with ‘Resnet.fc = nn.Sequential(nn.Conv2d(512, 6000), nn.Linear(6000,6000))’

1 Like
#8

Using this approach you would have to define a Flatten layer, since it will crash in the Sequential model.

class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()
        
    def forward(self, x):
        x = x.view(x.size(0), -1)
        return x
        

seq = nn.Sequential(nn.Conv2d(512, 6000, 1),
                    Flatten(),
                    nn.Linear(6000,6000))

x = Variable(torch.randn(1, 512, 1, 1))
seq(x)
1 Like
#9

@ptrblck and @wangchust,

Thank you both.

#13

I find this very useful to create a custom model which inserts a new layer in middle of ResNet, but how do we get the pretrained weights? This is giving me a new neural net which doesn’t have the pre trained weights.
Thank you.

#14

I want to add attention layers in the pretrained resnet model how can I do so after every resnet block in the model.

#15

This is what I did, I think it works.

vgg11.features[0] = nn.Sequential(inserted_layer, vgg11.features[0])

I checked both parameters, and I think the pretrained parameters are preserved.

#16

I first divided the model into two parts; 1st half are the layers before the layer you want to add and the 2nd half contains the layers after your layer. So something like this:

encoder = nn.Sequential(*list(m.children())[:8])
decoder  = nn.Sequential(*list(m.children())[8:])

Then I added the layers I need, get a list of the layers and append each layer in the 2nd layer into the 1st one. Like this:

tmp_1 = list(encoder.children())
tmp_2 = list(decoder.children())
for i in tmp_2:
    tmp_1.append(i)

And then model = nn.Sequential(*list(tmp_1))

2 Likes
#17

Sir, In this, I would like to use the pre-trained weights up to layer4. And I have to do transfer learning for the remaining layers.

My requirement is
// considering pretrained model
from torchvision import models
resnet50 = models.resnet50(pretrained = True)

//pruning the layers
prune.l1_unstructured(resnet50.conv1, name=‘weight’, amount=0.4)

// adding layer in the middle (after layer 4)
class MyResNet50(models.ResNet):

def __init__(self, my_pretrained_model):

    super(MyResNet50, self).__init__(my_pretrained_model)
    self.pretrained_model = my_pretrained_model
    self.conv_feat = nn.Conv2d(in_channels=512, 
                               out_channels=512, 
                               kernel_size=3, padding=1)
    self.my_new_layers = nn.Sequential(nn.Linear(1000, 512),nn.Linear(512,1))
    
def forward(self, x):

    x = self.pretrained_model(x)
    x = self.conv_feat(x)
    x = self.avgpool(x)
    x = self.my_new_layers(x)

    return x

//creating object for modified resnet50
myextendedmodel = MyResNet50(resnet50)

sir, is it possible to get modified resnet50?

#18

Would this mean the overall model architecture stays the same as in the original ResNet while you are only training some layers (starting from layer5)? If so, you wouldn’t have to add new layers but would need to freeze the initial 4 layers by setting the .requires_grad attribute of the corresponding parameters to False.

#19

Thank you, sir for the solution.