How to delete last linear layer of ViT-H or how to replace it with Identity()?

hasan11085 · August 17, 2023, 12:38pm

Hello there! I’m trying to figure out how I could possibly replace or delete the last layer of VIT-H or ViT-b.
Here deleting children method doesn’t work like Resnet.

Thanks in advance for the suggestion!

hasan11085 · August 17, 2023, 2:07pm

looking for an answer!

ptrblck · August 17, 2023, 2:09pm

You can directly replace layers with an nn.Identity module.
I would also not recommend bumping the thread after 2h of wait time, as it’s just spamming the board and not helpful.

hasan11085 · August 17, 2023, 2:18pm

I can access to model.encoder layers. But I didn’t find any solution how to access the last linear layer in VIT!

danielmao2019 · February 6, 2024, 6:58pm

If you are loading the vision transformer from torchvision, try modifying the linear layer by model.heads = torch.nn.Identity(). If you check the source code (I’m looking at version 0.15.2), you’ll see that the output of model.encoder is passed to model.heads.

In general, you can find out the attributes of a Python object by obj.__dict__.