How to change transfer learning model for a lot of classes

Hello. I use efficient net v2 model pre-trained on imagenet-1k for my task, but I have around 4k classes. Feature block has 1280 outputs, and I think it is a moment, that ruined my results.
I found imagenet 22-k, but I can’t find how experts change last layers for the task with so many classes.
I just change last classifier block to one linear layer, and get around 30% of accuracy. Do you know, what I can do in this situation?

Here is an example for transfer-learning from pytroch documentation.
In general, you should change the size of last classification layer. And retrain the model.
Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.13.1+cu117 documentation

Yes, I read this documentation. I guess I have problem with bottleneck.
Because feature block has 1280 outputs, but I need classify 4 000 classes

You can alter the particular layer of pretrained model once it is loaded.

Refer this answer for more information.

And if the accuracy is lower, you can add some layers before the last one and train it few more epochs by freezing the pretrained layers.

Hope it helps!.