Using pretrained 3D ResNet 50 without classifier layer

Hello,

I am using 3D ResNet 50 in PyTorch for the first time and have loaded the model pretrained on Kinetics using:

model = torch.hub.load(‘facebookresearch/pytorchvideo’, ‘slow_r50’, pretrained=True)

I want to be able to extract the “video features” calculated by the model for a given input (i.e. the outputs of the last layer before the linear and pool layers that calculate the Kinetics classification). However, when I do the following, I get an output size that is the same as my input size instead of a feature vector:

model = torch.hub.load(‘facebookresearch/pytorchvideo’, ‘slow_r50’, pretrained=True)
modules=list(model.children())[:-2]
model = nn.Sequential(*modules)
input = torch.ones(1, 3, 8, 256, 256) #dummy input
print(model(input).shape)

What is the correct code for creating a new 3D ResNet 50 model with the pretrained weights that doesn’t include the last two layers? Since this model seems to have many embedded ModuleLists, I think I’m not being able to index into the right layers.

Thanks!

Trying to recreate a model by wrapping its internal modules into an nn.Sequential container assumes that model.children() returns modules in the exact same order they were used in the forward pass and that the actual model uses a strict sequential execution of these modules without e.g. conditions, multiple paths, concatenations etc. Both assumptions could of course easily break if you are not careful and check the implementation of the model.
Instead of trying to recreate the model you could try to replace the unwanted layers with nn.Identity modules, which would keep the original forward pass, but would just return the activations for these layers.

1 Like

Thanks so much! My bad for asking a potentially trivial next question, but what would be the right way to replace the last two layers in the model? The layers I want to replace are in this construction:

(5): ResNetBasicHead(
(pool): AvgPool3d(kernel_size=(8, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
(dropout): Dropout(p=0.5, inplace=False)
(proj): Linear(in_features=2048, out_features=400, bias=True)
(output_pool): AdaptiveAvgPool3d(output_size=1)
)

I would just want to change the proj and output_pool layers to identity layers.

Access the desired attributes of the model and assign new nn.Identity layers to these.
Something like this should work:

model[5].proj = nn.Identity()
model[5[.output_pool = nn.Identity()

assuming (5): ResNetBasicHead can be accessed via model[5].

1 Like

Hmm… the strange thing is that the model isn’t subscriptable:

model[5].proj = torch.nn.Identity()

Traceback (most recent call last):
File “”, line 1, in
TypeError: ‘Net’ object is not subscriptable

You should check the model architecture to see how modules are registered.
If I print the model I see:

Net(
  (blocks): ModuleList(
    (0): ResNetBasicStem(
      (conv): Conv3d(3, 64, kernel_size=(1, 7, 7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
      (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
      (pool): MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=[0, 1, 1], dilation=1, ceil_mode=False)
    )
    (1): ResStage(
      (res_blocks): ModuleList(
        (0): ResBlock(
          (branch1_conv): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
          (branch1_norm): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (branch2): BottleneckBlock(
...

so, this would work:

model.blocks[5].proj = nn.Identity()
model.blocks[5].output_pool = nn.Identity()
1 Like

Ah I see what you’re saying! That ended up working - thanks so much for your help!