How do you properly load weights while training?

vision
4

Interesting, I started digging into backing up the optimizer as well because I wasn’t. But even then, its still not working.

If I asked you to revert back to how a model was at epoch x how would you do it?

I was even thinking about emptying the gpu and re-adding the model with the state-dict because it works perfectly when I restart my notebook but not when load the state dict while I’m training.

Even if I do something like:

if epoch == 50:
   new_model = models.resnet18(pretrained=True)
   new_model.load_state_dict(torch.load('weights_from_epoch_40'))
   new_model.to(DEVICE)

Then epoch 51 will have the same accuracy as if I never loaded the weights from epoch 40