I have saved and loaded the checkpoints as per the pytorch manual, and it all seems OK. Now, usually, when I want to start training, I have something like this in pytorch:
for itr in range(1, args.niters + 1):
optimizer.zero_grad() # should I or should I not?
I am unsure if I should do zero_grad()
here (which I use when I start training from scratch), since I am reloading all my weights and bias.
Apologies if this is a daft question.