Freezing/unfreezing model before/after mounting on gpu

Must freezing/unfreezing model be done necessarily before mounting it on gpu or can it be done whenever?

Hi,

Depends how you freeze your model. But I would recommend doing it last. As in general, moving to the GPU will create new Tensors and some of the metadata can be lost.

Sorry @albanD , I didn’t quite understand. Is mounting on gpu recommended to be done last or the contrary?

Moving to the gpu is recommended to be done first.
For example, it should be done before passing the parameters to the optimizer as well.

1 Like

I’d like to ask what you meant by “it depends on how the model is frozen”.

You can freeze the model in multiple ways:

  • set the requires_grad field to False
  • run in no_grad mode because you don’t need gradients to flow through
  • just never call optimizer.step()
    etc

The ones that rely on the parameters not to change might not work if you do them before sending the model to cuda. The ones that do not touch the parameters (2 and 3 above), will work either way :slight_smile:

1 Like