Resume training from checkpoints produces different losses

[pinocchio (Rene Sandoval)]

did you ever solve this issue?

---

related:

Resume iterating dataloader from checkpoint batch_idx

Hi, I was wondering whether it is possible to resume iterating through a dataloader from a checkpoint. For example: dataloaders_dict = {phase: torch.utils.data.DataLoader(datasets_dict[phase], batch_size=args.batch_size, num_workers=args.num_workers, shuffle=False) for phase in ['train']} # make sure shuffling is false incase you restart if os.path.isdir(args.batchidx_checkpoint): checkpoint = torch.load(args.batchidx_checkpoint + 'batches.pt') batch_idx = checkpo…

Resuming from model checkpoints produces different training loss

Hello, for the last 2 days I am trying to solve issue when resuming training from model checkpoint. Problem is that the training loss after resuming is a LOT different than before saving model (the difference is huge, almost as if the model was right after initialization process). I can see, that after few iterations it increases accuracy (decreases loss) much faster than if it was learned from scratch. But I don’t know what cuases the loss to be so high. But weird thing is that I checked outp…