# Adding intermediary layers to a pre-trained (BERT) model

**URL:** <https://discuss.pytorch.org/t/adding-intermediary-layers-to-a-pre-trained-bert-model/158825>\
**Category:** Uncategorized\
**Created:** [August 10, 2022, 12:24pm UTC](https://discuss.pytorch.org/t/adding-intermediary-layers-to-a-pre-trained-bert-model/158825 "2022-08-10T12:24:13Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 10, 2022, 10:00pm UTC](https://discuss.pytorch.org/t/adding-intermediary-layers-to-a-pre-trained-bert-model/158825/2 "2022-08-10T22:00:29Z")

</div>

> [@Antti](#):
>
> The most promising method I’ve found would seem to be as described here: [How can I replace the forward method of a predefined torchvision model with my customized forward function? - #8 by Shisho\_Sama](https://discuss.pytorch.org/t/how-can-i-replace-the-forward-method-of-a-predefined-torchvision-model-with-my-customized-forward-function/54224/8).

I would not try to monkey-patch the `forward` method as I assume it can break easily.

Instead, I would load the pre-trained model, make sure all parameters are properly loaded, and then manipulate the model by replacing a pre-trained layer with an `nn.Sequential` block containing the original pre-trained layer as well as the new one.  
E.g. something like this might work:

```python
model = MyModel(pretrained=True)
my_new_layer = NewLayer()

my_original_layer = copy.deepcopy(model.my_layer)
model.my_layer = nn.Sequential(
    my_original_layer,
    my_new_layer
)

```

---

_[View the full topic](https://discuss.pytorch.org/t/adding-intermediary-layers-to-a-pre-trained-bert-model/158825)._