Fine-tuning a middle layer

autograd
1

I’m trying to fine-tune a model with several layers. All the examples I’ve seen about fine-tuning is about the last layer or the last few layers, and it’s done by setting requires_grad=False on other layers.

If instead, I want to fine-tune a middle layer, can I just do the same by setting requires_grad=False for other layers? I’m wondering because in my understanding, requires_grad=True on an earlier layer will also cause requires_grad=True for later layers. I’m not sure if this is correct.

2

You can just set requires_grad=False on other layers. It would work exactly as you intended.

3

Thanks! So in general, is it true that only the params with requires_grad=True will be updated?

4

Yes :slight_smile: The optimizers look at the .grad attributes. If those are not there, optimizers do nothing to the parameters.