How does ResNet residual layer get preserved when converted to Sequential?

vision
1

I am using first two blocks of resnet50 in the following way

from timm.models.resnet import resnet50
r50 = resnet50(pretrained=True)
2_layers = torch.nn.Sequential(*list(r50.children())[:-4])

My thought was that since the blocks are now in Sequential, the residual part won’t be used and the output of the first 2 blocks of r50 and the final output of 2_layers should be different. However, the output is the same. Can someone explain how is the residual part retained when the layers are converted to Sequential? Link to code would be useful. Thanks in advance!

2

The skip connections are defined inside of self contained Modules (Bottleneck & BasicBlock). Since they are done in these modules, they are kept.

If the skip connections were done in the forward pass of the actual ResNet class, then they would not be kept. Here is the documentation for resnet from timm, directed to the forward method of the resnet class.

Here is a little dumm example to show what I mean.

The basic block adds 2 to the input in an extra function.
The ResN block does exactly the same, adding 5 instead.

When I run a tensor with ones through the whole model, I get 8 in every value as expected (1+2+5).
If I now do a feature extraction and put them in a nn.Sequential as you suggested, the result is 3.

It is not the same as the whole module.
The +5 from ResN is gone.
However, the +2 from the block is kept.

Hope this helps.

import torch


class BasicBlock(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def _functional_API(self, x):
        return x + 2

    def forward(self, x):
        x = self._functional_API(x)
        return x


class ResN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.blk = BasicBlock()

    def _functional_API(self, x):
        return x + 5

    def forward(self, x):
        x = self._functional_API(x)
        return self.blk(x)


model = ResN()
feature_extraction = torch.nn.Sequential(*list(model.children()))

tensor = torch.ones(5, 5)

print(model(tensor))
print(feature_extraction(tensor))

# Output:
tensor([[8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.]])
tensor([[3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.]])
2 Likes
3

The layer of ResNet is Converted to Sequential but there is a custom froward function for the ResNet block that keeps the residual part

def forward(self, x: Tensor) -> Tensor:
        identity = x

         ......

        out += identity # add residual part
        out = self.relu(out)

        return out

full code is available at
https://pytorch.org/vision/stable/_modules/torchvision/models/resnet.html#resnet50