How to Unfreeze Layers one by one

1

Hi!

I’m working with a PyTorch model from here (T2T_ViT_7).

I’m trying to freeze all layers except the last (head) layer and train it on my dataset. I want to evaluate its performance, and then unfreeze layers one by one and train it each time and see how it performs.

To initially freeze all the layers and and just unfreeze the head layer, I used:

for param in model.parameters():
    param.requires_grad_(False)

model.head.requires_grad_(True)

Now I want to start from the bottom, and start unfreezing layers one by one. How can I do this? Do I use model.modules() or maybe model.children()?

Thank you!

2

You just can’t.
Think that a model is some sort of tree. Each module has many submodules but only a single parent. The parameters are obtained recursively exploring all the childrends but the root is not aware of how many childrens and subchildrens are there.

You can just store the object in a list and read from the “end” (there is no such thing) to the root.