Inserting custom layer in a pretrained model

[srishti-git1110 (Srishti Gureja)]

New layer in BERT

You can simply replace the 6th layer with another layer. This layer will have 6th layer encoder as first layer and your custom layer as the second layer. For example, let’s try adding another encoder in the 5th layer:- from torch import nn from transformers import BertForSequenceClassification, BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertForSequenceClassification.from_pretrained('bert-base-uncased') class CustomBertLayer(nn.Module): def __init_…

Does this help?
It’s not about GPT-2, but maybe you could replicate this for your use case.