Reusing optimizer after model deletion

1

Hello,

Suppose I have some model (nn.Module) and an associated optimizer to model, e.g., model = optim.SGD([param for param in model.parameters()], lr=LR) and an lr_scheduler=optim.lr_scheduler.MultiStepLR(optimizer, milestones=MILESTONES, gamma=GAMMA)

Suppose I want to free memory for some time, to do some other tasks (e.g., train some other models), then reload the same model to the memory while keeping track of optimizer and lr_scheduler state.

My questions are

  • what is the best way to delete model is it del model or should I do do for all param in model.parameters()?
  • if I delete the model, the values of its parameters are still held by the optimizer, so I need to delete them also from the optimizer, in that case I will need to delete those also to free memory, what is the correct way of doing this, is it to delete all the parameters for param_group in optimizer.param_groups?
  • How to relaod the recreate the model when needed, should I recreate a new optimizer and a new lr_scheduler or can I just add param_groups to to the optimizer?